比如替换///成:::
可以用sed 's/\/\/\//:::/g' 即在前面加一个反斜杆
当然这样不友好
我们更喜欢
sed '_///_:::_g' 即把分隔符换了 换成其他符号如: 都可以的
替换一长串莫名字符，比如把以下内容//到:之间都替换成@
//abcd19388102738234:
//ijsh12398097324:
//pppppafasfae1231231:
可以用sed 's_//.*:_@_g'
星期一, 十月 26, 2009
星期二, 五月 26, 2009
sed学习笔记
★ 命令行参数简介
sed
-e script 指定sed编辑命令
-f scriptfile 指定的文件中是sed编辑命令
-n 寂静模式，抑制来自sed命令执行过程中的冗余输出信息，比如只
显示那些被改变的行。
不明白？不要紧，把这些肮脏丢到一边，跟我往下走，不过下面的介绍里
不包括正则表达式的解释，如果你不明白，可能有点麻烦。
★ 首先假设我们有这样一个文本文件 sedtest.txt
cat > sedtest.txt
Sed is a stream editor
----------------------
A stream editor is used to perform basic text transformations on an input stream
--------------------------------------------------------------------------------
While in some ways similar to an editor which permits scripted edits (such as ed
)
,
--------------------------------------------------------------------------------
-
-
sed works by making only one pass over the input(s), and is consequently more
-----------------------------------------------------------------------------
efficient. But it is sed's ability to filter text in a pipeline which particular
l
y
--------------------------------------------------------------------------------
-
★ 输出指定范围的行 p other types of editors.
sed -e "1,4p" -n sedtest.txt
sed -e "/from/p" -n sedtest.txt
sed -e "1,/from/p" -n sedtest.txt
★ 在每一行前面增加一个制表符(^I)
sed "s/^/^I/g" sedtest.txt
注意^I的输入方法是ctrl-v ctrl-i
单个^表示行首
★ 在每一行后面增加--end
sed "s/$/--end/g" sedtest.txt
单个$表示行尾
★ 显示指定模式匹配行的行号 [/pattern/]=
sed -e '/is/=' sedtest.txt
1
Sed is a stream editor
----------------------
3
A stream editor is used to perform basic text transformations on an input stream
--------------------------------------------------------------------------------
While in some ways similar to an editor which permits scripted edits (such as ed
)
,
--------------------------------------------------------------------------------
-
-
7
sed works by making only one pass over the input(s), and is consequently more
-----------------------------------------------------------------------------
9
efficient. But it is sed's ability to filter text in a pipeline which particular
l
y
--------------------------------------------------------------------------------
sort学习笔记
sort默认是把整行当作一串字符，以首字符排序
------------------------
不指明的sort是对整行的排序：
xyb 0$ sort a
a 9 t
c 2 f
d 4 x
Pasted from <http://www.unixresources.net/linux/clf/vrml/archive/00/00/34/71/347190.html>
------------------------
Sort -nk 5
以第5列进行排序
-n表示numberic排序
-k表示位置??
-r reverse 降序排列
+1 对第二列排序，如：（字段从0开始数，第一字段是0）
-----------------------------------------
按数字顺序排序第二个字段：
xyb 0$ sort -n +1 a
c 2 f
d 4 x
a 9 t
Pasted from <http://www.unixresources.net/linux/clf/vrml/archive/00/00/34/71/347190.html>
-------------------------------------------
两个字段排序：
-------------------------------------------
先按第二个、不按第三个字段排序： （这个解释估计错了，应该是第二字段升序，第三字段降序）
xyb 0$ sort +1 -2 a
c 2 f
d 4 x
a 9 t
d 9 c
+POS1 [-POS2]属于比较老的用法，现在gnu sort里推荐使用-k POS1[,POS2]，
表示排序关键字从POS1开始，到POS2中止。详细信息参考man sort和info sort
Pasted from <http://www.unixresources.net/linux/clf/vrml/archive/00/00/34/71/347190.html>
-------------------------------------------
-t 指定分隔符
-------------------------------------------
xyb 0$ sed 's/ /$/g' a | sort -t$ -n +1
c$2$f
d$4$x
a$9$t
Pasted from <http://www.unixresources.net/linux/clf/vrml/archive/00/00/34/71/347190.html>
-------------------------------------------
HPUX上Oracle 11g client安装笔记
OS：HPUX
hardware：安腾CPU
安装对象：
Oracle 11.1.0.6 client
sqlldr,sqlplus
Oracle 11.1.0.7 patch
sqlldr和sqlplus安装后，文件在$ORACLE_HOME/bin 下面
runtime安装会自动包含sqlplus
sqlldr在custome安装下面的oracle database utilities组件里，有其他东西，据说蛮有用，都装了
admin安装会包含sqlldr
安装界面出来，welcome下面有一步Inventory Directory,是临时安装文件存放的地方。保持这个路径为空或者删掉最后一级路径都可以。每次安装oracle都会重新copy内容进去的。
sqlldr起不起来的原因：
1. /u01/app/oracle/product/11.1.0.6/client_1/bin 这个路径要有rrr权限
2. sqlldr文件要有xxx权限
3. ORACLE_HOME要设进.profile
查看是否安腾处理器方法：
machinfo（Itanium机器可用）或者 ioscan，dmesg，model（非Itanium机器）
用unix连接XWindow以显示图形界面：
1.打开软件ReflectionX或者Xwindow软件，不要任何设置或者连接
2.export DISPLAY=16.100.129.5:0.0（在unix上直接输入这段命令，红色部分填你的电脑IP，这样执行runInstaller的时候unix会去连你电脑上的ReflectionX）
3.在安装文件解压的路径下：
cd client
./runInstaller -ignoreSysPrereqs
安装7patch的时候
cd Disk1
./runInstaller
星期四, 三月 26, 2009
archive log space
今天遇到这个问题：
SQL*Loader-128: unable to begin a session
ORA-00257: archiver error. Connect internal only, until freed.结果是archive log空间不够了。找DBA clear一下就行了。
星期三, 三月 25, 2009
zz Sqlldr的使用
http://blog.oracle.com.cn/index.php/266780/viewspace-26695
Sqlldr也就是SQL*LOADER，它是oracle的高速批量数据加载工具，可以将外部文件的数据导入到oracle的数据库中。可以用于从多种平面文件格式向oracle数据库中加载数据。
它有两种操作模式：传统路径（conventional path）：利用sql插入为我们加载的数据。直接路径（direct path）：不使用sql，而是直接格式化数据库块。利用直接路径加载，能从一个平面文件读取数据，并将其直接写至格式化的数据库块，而绕过整个sql引擎和undo生成，同时还可能避开redo生成，要在一个没有任何数据的数据库中充分加载数据，这是最好的方法。
SQLLDR包括五个文件:控制文件（*.ctl）、数据文件（*.dat）、日志文件(*.log)、错误文件(*.bad)、废弃文件(*.dsc)。
其中我们最常用的为前四中：导入方式为：
Sqlldr userid=user/password@SID control＝path\xx.ctl log=path\xx.log bad=path\ xx.bad
注意SID就是数据库名，path就是路径名，不要加’’，一般不用加数据文件，因为在control就包含了数据或者数据路径，log是需要的，它可以告诉你导入的详细信息，而bad则存放者错误的文件，discard（废弃文件）则存放者不满足导入条件的数据。
控制文件它的作用是告诉oracle如何的读取和加载信息，并提供数据的路径，加载的方式以及加载的规则的。具体格式如下“
Load data
Infile‘数据路径’
Into table table_name
Truncate
Fields terminated by‘’
（column_name1，column_name2，······）
具体含义：1、Load data的意义就是说载入数据2、Infile‘数据路径’就是指明了载入数据的位置，其中数据默认的扩展名是.dat，我们也可以更改比如.txt、.csv（逗号分割值形式）都可以，还有这里不仅可以导入数据文件，也可以导入错误文件或者废弃文件，格式如：badfile‘path\xx.bad’或者discardfile‘path\xx.dsc’3、Into table table_name把数据导入到的什么表，这里可以插入多个表，用when condition条件分割就可以了4、Truncate它的含义是当sqlldr执行这个控制文件是，表在开始加载前就给截断了。用Truncate是不能回退的一定要谨慎，除此之外，还可以使用append（用于在表中增加行）、insert（用于在空表中增加行，如果不为空，加载就会错误）、replace（用于清空表，然后在增加新行，但是用户必须有该表的delete权限），前提注意，执行加载的用户必须具有表的insert权限，如果没有该参数，系统默认的为insert。5、Fields terminated by‘’数据的分割符，首先先要明白sqlldr加载数据我们常用的有两种装载定长数据和装载变长数据。例子如下：
装载定长数据
load data
infile ‘xx.txt’
into table table_name append
(column_name1 position(01:10) character,
column_name2 position(11:12) character,
column_name3 position(13:14) character,
column_name4 position(15:16) character,)
它是装载了未知固定好的数据。
装载定长数据
load data
infile ‘x.dat’
into table table_name append
(column_name1 char terminated by ‘ ‘,
column_name2 char terminated by ‘ ‘,
column_name3 char enclosed by ‘ ‘,
column_name4 char terminated by whitespace)
或者
load data
infile ‘x.dat’
into table table_name append
Fields terminated by“”
(column_name1,column_name2，····)
这里注意一下Fields terminated by‘’，里面的‘’表示分割符号，比如说‘/’、‘；’,也可以FIELDS TERMINATED BY X‘09’（制表符）, FIELDS TERMINATED whitespace（空格）等等，还可以在后面加上OPTIONALLY ENCLOSED BY“”，它的意思实说每个选定的字段是用“”表示的。LINES TERMINATED‘\t‘ 表示每行记录之间用什么分隔默认的为\n（可以不加的）6、（column_name1，column_name2，······）就是把数据加入到表中的哪些字段。7、还可以加一个选项trailing nullcols指没有的数据用NULL填充。
我们在看一个有选择条件的加载多个表方式
Load data
infile 'x.txt'
replace into table table1
when column_name = 'condition1'
(column_name1，column_name2，····)
when column_name =‘condition
into table table2
(column_name1，column_name2，····)
导入EXCEL，可以把EXCEL文件另存为CSV（逗号分隔）(*.csv),控制文件就改为用逗号分隔
LOAD DATA
INFILE 'x.csv'
APPEND INTO TABLE table
FIELDS TERMINATED BY ","
(column_name1, column_name2，····)
使用filler跳过在导入数据文本中不想进行导入的列。在控制文件中还可以使用concatenate语句将多个物理行合成一个逻辑行插入到表中，在文件中加注释是用――后根语句来注释的。理想的情况下我们加载数据都希望不是完全的成功就是完全的失败这样便于第二次加载，但是实际上往往是部分成功和部分失败，一次要利用日志文件判断出失败的地方，日志文件会记录提交点和错误产生点。所有被拒绝的记录保存在坏文件和废弃文件中，因此我们必须活用sqlldr的选现来实现数据的加载。
如果加载的数据过多，我们可以作一个.bat的批处理文件把所有的sqlldr加载语句放入其中，最后直接执行批处理文件就可以。
最后一个就是并行并发操作：比如
sqlldr userid=/ control=result1.ctl direct=true parallel=true
sqlldr userid=/ control=result2.ctl direct=true parallel=true
sqlldr userid=/ control=result2.ctl direct=true parallel=true
当加载大量数据时（大约超过10GB），最好抑制日志的产生：
SQL>ALTER TABLE RESULTXT nologging;
这样不产生REDO LOG，可以提高效率。然后在CONTROL文件中load data上面加一行：unrecoverable（不可恢复）此选项必须要与DIRECT共同应用。
因为sqlldr在加载数据时会产生大量的insert语句，因此使用direct path先格式化数据快，在把数据块快数插入表中。提高性能。使用了direct path就可以使用unrecoverable关键字提高数据加载性能。可以不必生成重做日志项。同时可以使用parallel direct path加载选项将数据加载工作分为多个进程。因此并行的direct path加载操作比单个的direct path加载效率快很多。
Sqlldr选项参考工作图中的sqlldr.jpg
另外还要注意一个使TRAILING NULLCOLS，因为在引导数据中，比如说你向往5列中引入数据，但数据每行只有4个，而第5列之前前四行之和，但是在处理中sqlldr会告诉你，没有等处理完所有的列，记录中就没有数据了。因此我们就必须使用TRAILING NULLCOLS，这样一来，如果数据记录中不存在某列的数据，sqlldr就会先为该列绑定一个null值，因此我们执行第5列之前，实际上它的值就使null，而不会报错了。（注意，在加载数据时可以使用sql连接运算符）
加载有内嵌换行符的数据，一种方法可以用非换行符的其他字符来表示换行符比如说在文本中应该出现换行符的位置上放上一个\n，并在加载时使用一个SQL函数用以chr（10）来替换该文本。二种是使用FIX属性，但是这种方法，输入数据必须出现在定长记录中，每个记录与输入数据集中所有其他记录的长度都相同，既有相同的字节数。使用FIX属性，必须使用一个INFILE子句，因为FIX是INFILE一个选项，而且，数据必须在外部存储，而并非存储在控制文件本身，如：INFILE TEST.DAT “FIX 80”指定一个输入数据文件，这个文件每个记录都有80个字节，包括尾部的换行符。在这种情况下，输入文件中提供给sqlldr的记录设置就以非\n结束的。第三种情况，可以使用VAR属性，这种格式，每个记录必须以某个固定的字节数开始，这标识这个记录的总长度。可以加载包含内嵌换行符的变长记录，但是每个记录的开始处必须有一个记录长度的字段。如：infile test.dak“var 3”这里指出了每个输入记录的前3个字节是该输入记录的长度。还有一种情况是使用STR属性，这算是最灵活的一种，可以指定一个新的行结束字符（或字符序列）。这样就能创建一个输入数据文件，其中每一行的最后有某个特殊字符，换行符不再有特殊含义。STR属性是以16进制指定的，因此要得到所需的具体16进制串，最容易的办法是使用SQL和UTL_RAW来生成16进制串,可以直接导入在unix平台下的数据。总结一下：要注意在windows和unix两个平台的不同，结束符是不同的unxi上是\n，而windows上是\r\n，\r是记录的一部分，控制文件必须适应这一点，比如，如果取一个.dat文件只包含\n当传输到windows上是，要将各个\n转换为\r\n。那么原来unix种的控制文件就不能记载数据了。
ll tips
一、在当前文件夹ll结果仅有文件时，不会有total number的第一行
二、在当前文件夹ll结果含有文件夹时，会有如total 100的第一行，且包含文件夹中的文件数量
三、ll的结果中，时间不在当年的，会显示年份；但在当年的，会显示时刻，如：
-rw------- 1 ncrfmgr ncrf 0 Oct 8 15:28 test1.txt
-rw-rw-r-- 1 ncrfsftp ncrf 1934 Sep 9 2008 Update_Amidmaster_20080908_TW.txt
可能的解决方法（待解决）
1.修改set，LC_TIME=POSIX改成ISO或者ISO2 （时间的格式参见man ls）
2.在非HP-UX的系统中，可以尝试ls -l --time-style=iso这样的做法，在man ls里有详细介绍。但是部分unix版本不支持。
3.强制结果把含有冒号：的改成当年年份，缺点是做法太麻烦。（顺便查一下如果查找if字符串中含有符号的做法）
四、ll 一个路径，结果文件名中只显示文件名；但是ll一个路径+文件名，结果的文件名会带上路径。如：
$ ll /var/opt/ncrf/mkt/inbound/bulkload/amidmaster
total 2
-rw------- 1 ncrfmgr ncrf 104 Mar 25 09:54 amidmaster_list.txt
drwx------ 2 ncrfmgr ncrf 96 Mar 25 08:00 archive
-rw------- 1 ncrfmgr ncrf 0 Mar 25 09:51 temp.txt
-rw------- 1 ncrfmgr ncrf 0 Mar 25 09:18 testfile
$ ll /var/opt/ncrf/mkt/inbound/bulkload/amidmaster/amidmaster_list.txt
-rw------- 1 ncrfmgr ncrf 104 Mar 25 09:54 /var/opt/ncrf/mkt/inbound/bulkload/amidmaster/amidmaster_list.txt
五、ll文件找不到时，会有一行"*** not found"的结果 ***为文件名。
星期一, 三月 23, 2009
zz Unix Shell中引号的用法
1. 单引号 ( ' ' )
howard@0[script]$ grep Susan phonebook
Susan Goldberg 403-212-4921
Susan Topple 212-234-2343
如果我们想查找的是Susan Goldberg，不能直接使用grep Susan Goldberg phonebook命令，grep会把Goldberg和phonebook当作需要搜索的文件
howard@0[script]$ grep 'Susan Gold' phonebook
Susan Goldberg 403-212-4921
当shell碰到第一个单引号时，它忽略掉其后直到右引号的所有特殊字符
2. 双引号 ( " " )
双引号作用与单引号类似，区别在于它没有那么严格。单引号告诉shell忽略所有特殊字符，而双引号只要求忽略大多数，具体说，括在双引号中的三种特殊字符不被忽略：$,\,` ,即双引号会解释字符串的特别意思,而单引号直接使用字符串.如果使用双引号将字符串赋给变量并反馈它，实际上与直接反馈变量并无差别。如果要查询包含空格的字符串，经常会用到双引号。
howard@0[script]$ x=*
howard@0[script]$ echo $x
hello.sh menus.sh misc.sh phonebook tshift.sh
howard@0[script]$ echo '$x'
$x
howard@0[script]$ echo "$x"
*
这个例子可以看出无引号、单引号和双引号之间的区别。在最后一种情况中，双引号告诉shell在引号内照样进行变量名替换，所以shell把$x替换为＊，因为双引号中不做文件名替换，所以就把＊作为要显示的值传递给echo。
对于第一种情况需要进一步说明，shell在给变量赋值时不进行文件名替换（这从第三种情况中也能看出来），各步骤发生的精确次序如下：
shell扫描命令行，把x的值设为星号＊；
shell再次扫描命令行，碰到星号＊，把它替换成当前目录下的文件清单；
shell启动执行echo命令，把文件清单作为参数传递给echo.
这个赋值的先后次序非常重要：shell先作变量替换，然后作文件名替换，最后把这行处理为参数
shell再次扫描命令行，碰到星号＊，把它替换成当前目录下的文件清单；
shell启动执行echo命令，把文件清单作为参数传递给echo.
这个赋值的先后次序非常重要：shell先作变量替换，然后作文件名替换，最后把这行处理为参数
3. 反引号(``)
命令替换是指shell能够将一个命令的标准输出插在一个命令行中任何位置。shell中有两种方法作命令替换：把shell命令用反引号或者$(...)结构括起来，其中，$(...)格式受到POSIX标准支持，也利于嵌套。
howard@0[script]$ echo The date and time is `date`
The date and time is 三 6月 15 06:10:35 CST 2005
howard@0[script]$ echo Your current working directory is $(pwd)
Your current working directory is /home/howard/script.
4. 反斜杠 backslash-escaped( \ )
反斜杠一般用作转义字符,或称逃脱字符,linux如果echo要让转义字符发生作用,就要使用-e选项
星期日, 三月 22, 2009
LS用法
LSSection: User Commands (1)Updated: November 1998 NAMEls, dir, vdir - 列目錄內容提要ls [選項] [文件名...]GNU 選項 (短格式):[-1abcdfgiklmnopqrstuxABCDFGLNQRSUX] [-w cols] [-T cols] [-I pattern] [--full-time][--format={long,verbose,commas,across,vertical,single-column}] [--sort={none,time,size,extension}] [--time={atime,access,use,ctime,status}] [--color[={none,auto,always}]] [--help] [--version] [--]描述（ DESCRIPTION ）程序ls先列出非目錄的文件項，然是每一個目錄中的 “可顯示”文件。如果沒有選項之外的參數【譯注：即 文件名部分為空】出現，缺省為 "." （當前目錄）。選 項“ -d ”使得目錄與非目錄項同樣對待。除非“ -a ” 選項出現，文件名以“.”開始的文件不屬“可顯示”文 件。 以當前目錄為準，每一組文件（包括非目錄文件項，以及 每一內含文件的目錄）分別按文件名比較順序排序。如果 “ -l ”選項存在，每組文件前顯示一摘要行給出該 組文件長度之和（以 512 字節為單位）。 輸出是到標準輸出（ stdout ）。除非以“ -C ”選項要 求按多列輸出，輸出將是一行一個。然而，輸出到終端時， 單列輸出或多列輸出是不確定的。可以分別用選項“ -1 ” 或“ -C ”來強制按單列或多列輸出。
GNU 細節如果標準輸出是終端，將多列輸出（縱向排序）。
GNU 選項-1, --format=single-column 一行輸出一個文件（單列輸出）。如標準輸出不是到終端， 此選項就是缺省選項。
GNU 標準選項
環境變量 POSIXLY_CORRECT 可以決定一組選擇。如果沒有設置此變量，每個制表符的字 符數由變量 TABSIZE 決定。變量 COLUMNS （當它由一個十進制整數表示時）決定輸 出的列寬度（同“ -C ”選項一起用時）。文件名不得為適應多列輸出而被截斷。變 量 LANG, LC_ALL, LC_COLLATE, LC_CTYPE, LC_MESSAGES 及 LC_TIME 仍保持原義。 變量 TZ 給出時區供 ls 輸出相應的時間字串。變量 LS_COLORS 用以決定是否使用 顏色。已知錯誤在 BSD 系統上，對從 HP-UX 系統上通過 NFS mount 而來的文件，“ -s ”選項報 告的大小只有正確值的一半；在 HP-UX 系統上，對從 BSD 系統上通過 NFS mount 而來的文件， ls 報告的大小則有正確值的兩倍。這是 HP-UX 的一個缺陷造成的，它 也影響 HP-UX 上的 ls 程序。適合到POSIX 1003.2參見dircolors(1)注意本頁描述的是 fileutils-3.16 文件包中的 ls ，其它版本的可能略有不同。糾錯或添 加（功能）請 mailto: aeb@cwi.nl 和 aw@mail1.bet1.puv.fi 及 ragnar@lightside.ddns.org 。本程序的錯誤報告請 mailto: fileutils-bugs@gnu.ai.mit.edu 。[譯者]：wangdong@163.net[版權協議]：GNU's GPL 2.0 or higher日期： Oct. 7th, 2000 |
| 相關文章 |
| ls 命令選項詳解 (2001-04-15 19:37:13) |
星期三, 七月 18, 2007
UNIX bc的小数除法
这个东东整了我大半天的时间。。。
因为前面的代码算出一个TotalTime的结果是个科学计数法1.23456e+08这样的东东，在windows的MKS下只要简单的用
AverageTime=`expr $TotalTime / $total_num`
就ok了。
搬到UNIX服务器下跑就报expr expect a integer
开始是以为分子必须为整数，就搜出来一个方法，用bc控制分子的精度
AverageTime=`echo "scale=0;$TotalTime/$total_num"|bc' //这个对于分子是小数点的情况还是好使的：）
这个方法在UNIX命令行直接跑能赋值，但是echo $AverageTime就会报Syntex Error in line1,
代入数字，一行行跑：
AverageTime=`echo "scale=2;$2.5/$2"|bc'
echo $AverageTime
能得出1
然后找Jason帮我调了调，到代人
AverageTime=`echo "scale=1.23456e+08;$2.5/$2"|bc'
同样报出Syntex Error in line1,
最后一顿狂搜，快下班的时候终于让我找到打印命令，开始用%d，说无法转换，想起来以前C的时候也用一样的格式符，就试了%.0f，搞定～
AverageTime=`expr $TotalTime / $total_num`
星期二, 七月 17, 2007
写shell script的基础篇zz
找了个写shell script的基础篇，zz一下：
Shell Programming
![[Illustration]](https://lh3.googleusercontent.com/blogger_img_proxy/AEn0k_uwYwZimLG-1Kde0qw7sPvr36vehv2e_rgxKs7_mzElJe9fXj3qdbmWEIhMNMH-QJAn0-suphK94siN3Yqb-SGtTqD1jav-5NO8LFrrJE6DIsLIFzDZkCCo8NUo5JhHWcXNl_viTgvHeDhHM6N9lQ8GZVHqOdA=s0-d)
Abstract:
In this article we explain how to write little shell scripts and give many examples.
Why shell programming?
Even though there are various graphical interfaces available for Linux the shell still is a very neat tool. The shell is not just a collection of commands but a really good programming language.You can automate a lot of tasks with it, the shell is very good for system administration tasks, you can very quickly try out if your ideas work which makes it very useful for simple prototyping and it is very useful for small utilities that perform some relatively simple tasks where efficiency is less important than ease of configuration, maintenance and portability.So let's see now how it works:
Creating a script
There are a lot of different shells available for Linux but usually the bash (bourne again shell) is used for shell programming as it is available for free and is easy to use. So all the scripts we will write in this article use the bash (but will most of the time also run with its older sister, the bourne shell).For writing our shell programs we use any kind of text editor, e.g. nedit, kedit, emacs, vi...as with other programming languages.
The program must start with the following line (it must be the first line in the file):
#!/bin/shThe #! characters tell the system that the first argument that follows on the line is the program to be used to execute this file. In this case /bin/sh is shell we use.
When you have written your script and saved it you have to make it executable to be able to use it.
To make a script executable type
chmod +x filename
Then you can start your script by typing: ./filename
Comments
Comments in shell programming start with # and go until the end of the line. We really recommend you to use comments. If you have comments and you don't use a certain script for some time you will still know immediately what it is doing and how it works.Variables
As in other programming languages you can't live without variables. In shell programming all variables have the datatype string and you do not need to declare them. To assign a value to a variable you write:varname=valueTo get the value back you just put a dollar sign in front of the variable:
#!/bin/shType this lines into your text editor and save it e.g. as first. Then make the script executable by typing chmod +x first in the shell and then start it by typing ./first
# assign a value:
a="hello world"
# now print the content of "a":
echo "A is:"
echo $a
The script will just print:
A is:Sometimes it is possible to confuse variable names with the rest of the text:
hello world
num=2This will not print "this is the 2nd" but "this is the " because the shell searches for a variable called numnd which has no value. To tell the shell that we mean the variable num we have to use curly braces:
echo "this is the $numnd"
num=2This prints what you want: this is the 2nd
echo "this is the ${num}nd"
There are a number of variables that are always automatically set. We will discuss them further down when we use them the first time.
If you need to handle mathematical expressions then you need to use programs such as expr (see table below).
Besides the normal shell variables that are only valid within the shell program there are also environment variables. A variable preceeded by the keyword export is an environment variable. We will not talk about them here any further since they are normally only used in login scripts.
Shell commands and control structures
There are three categories of commands which can be used in shell scripts:1)Unix commands:
Although a shell script can make use of any unix commands here are a number of commands which are more often used than others. These commands can generally be described as commands for file and text manipulation.
Command syntax
Purpose
echo "some text"
write some text on your screen
ls
list files
wc -l file
wc -w file
wc -c file
count lines in file or
count words in file or
count number of characters
cp sourcefile destfile
copy sourcefile to destfile
mv oldname newname
rename or move file
rm file
delete a file
grep 'pattern' file
search for strings in a file
Example: grep 'searchstring' file.txt
cut -b colnum file
get data out of fixed width columns of text
Example: get character positions 5 to 9
cut -b5-9 file.txt
Do not confuse this command with "cat" which is something totally different
cat file.txt
write file.txt to stdout (your screen)
file somefile
describe what type of file somefile is
read var
prompt the user for input and write it into a variable (var)
sort file.txt
sort lines in file.txt
uniq
remove duplicate lines, used in combination with sort since uniq removes only duplicated consecutive lines
Example: sort file.txt | uniq
expr
do math in the shell
Example: add 2 and 3
expr 2 "+" 3
find
search for files
Example: search by name:
find . -name filename -print
This command has many different possibilities and options. It is unfortunately too much to explain it all in this article.
tee
write data to stdout (your screen) and to a file
Normally used like this:
somecommand | tee outfile
It writes the output of somecommand to the screen and to the file outfile
basename file
return just the file name of a given name and strip the directory path
Example: basename /bin/tux
returns just tux
dirname file
return just the directory name of a given name and strip the actual file name
Example: dirname /bin/tux
returns just /bin
head file
print some lines from the beginning of a file
tail file
print some lines from the end of a file
sed
sed is basically a find and replace program. It reads text from standard input (e.g from a pipe) and writes the result to stdout (normally the screen). The search pattern is a regular expression (see references). This search pattern should not be confused with shell wildcard syntax. To replace the string linuxfocus with LinuxFocus in a text file use:
cat text.file | sed 's/linuxfocus/LinuxFocus/' > newtext.file
This replaces the first occurance of the string linuxfocus in each line with LinuxFocus. If there are lines where linuxfocus appears several times and you want to replace all use:
cat text.file | sed 's/linuxfocus/LinuxFocus/g' > newtext.file
awk
Most of the time awk is used to extract fields from a text line. The default field separator is space. To specify a different one use the option -F.
cat file.txt | awk -F, '{print $1 "," $3 }'
Here we use the comma (,) as field separator and print the first and third ($1 $3) columns. If file.txt has lines like: Adam Bor, 34, Indiathen this will produce:
Kerry Miller, 22, USA
Adam Bor, IndiaThere is much more you can do with awk but this is a very common use.
Kerry Miller, USA
2) Concepts: Pipes, redirection and backtick
They are not really commands but they are very important concepts.
pipes (|) send the output (stdout) of one program to the input (stdin) of another program.
grep "hello" file.txt | wc -lfinds the lines with the string hello in file.txt and then counts the lines.
The output of the grep command is used as input for the wc command. You can concatinate as many commands as you like in that way (within reasonable limits).
redirection: writes the output of a command to a file or appends data to a file
> writes output to a file and overwrites the old file in case it exists
>> appends data to a file (or creates a new one if it doesn't exist already but it never overwrites anything).
Backtick
The output of a command can be used as command line arguments (not stdin as above, command line arguments are any strings that you specify behind the command such as file names and options) for another command. You can as well use it to assign the output of a command to a variable.
The command
find . -mtime -1 -type f -printfinds all files that have been modified within the last 24 hours (-mtime -2 would be 48 hours). If you want to pack all these files into a tar archive (file.tar) the syntax for tar would be:
tar xvf file.tar infile1 infile2 ...Instead of typing it all in you can combine the two commands (find and tar) using backticks. Tar will then pack all the files that find has printed:
#!/bin/sh
# The ticks are backticks (`) not normal quotes ('):
tar -zcvf lastmod.tar.gz `find . -mtime -1 -type f -print`
3) Control structures
The "if" statement tests if the condition is true (exit status is 0, success). If it is the "then" part gets executed:
if ....; thenMost of the time a very special command called test is used inside if-statements. It can be used to compare strings or test if a file exists, is readable etc...
....
elif ....; then
....
else
....
fi
The "test" command is written as square brackets " [ ] ". Note that space is significant here: Make sure that you always have space around the brackets. Examples:
[ -f "somefile" ] : Test if somefile is a file.Run the command "man test" and you get a long list of all kinds of test operators for comparisons and files.
[ -x "/bin/ls" ] : Test if /bin/ls exists and is executable.
[ -n "$var" ] : Test if the variable $var contains something
[ "$a" = "$b" ] : Test if the variables "$a" and "$b" are equal
Using this in a shell script is straight forward:
#!/bin/shThe variable $SHELL contains the name of the login shell and this is what we are testing here by comparing it against the string "/bin/bash"
if [ "$SHELL" = "/bin/bash" ]; then
echo "your login shell is the bash (bourne again shell)"
else
echo "your login shell is not bash but $SHELL"
fi
Shortcut operators
People familiar with C will welcome the following expression:
[ -f "/etc/shadow" ] && echo "This computer uses shadow passwors"The && can be used as a short if-statement. The right side gets executed if the left is true. You can read this as AND. Thus the example is: "The file /etc/shadow exists AND the command echo is executed". The OR operator (||) is available as well. Here is an example:
#!/bin/shThe script tests first if it can read a given mailfolder. If yes then it prints the "From" lines in the folder. If it cannot read the file $mailfolder then the OR operator takes effect. In plain English you read this code as "Mailfolder readable or exit program". The problem here is that you must have exactly one command behind the OR but we need two:
mailfolder=/var/spool/mail/james
[ -r "$mailfolder" ] || { echo "Can not read $mailfolder" ; exit 1; }
echo "$mailfolder has mail from:"
grep "^From " $mailfolder
-print an error message
-exit the program
To handle them as one command we can group them together in an anonymous function using curly braces. Functions in general are explained further down.
You can do everything without the ANDs and ORs using just if-statements but sometimes the shortcuts AND and OR are just more convenient.
The case statement can be used to match (using shell wildcards such as * and ?) a given string against a number of possibilities.
case ... inLet's look at an example. The command file can test what kind of filetype a given file is:
...) do something here;;
esac
file lf.gzWe use this now to write a script called smartzip that can uncompress bzip2, gzip and zip compressed files automatically :
returns:
lf.gz: gzip compressed data, deflated, original filename,
last modified: Mon Aug 27 23:09:18 2001, os: Unix
#!/bin/sh
ftype=`file "$1"`
case "$ftype" in
"$1: Zip archive"*)
unzip "$1" ;;
"$1: gzip compressed"*)
gunzip "$1" ;;
"$1: bzip2 compressed"*)
bunzip2 "$1" ;;
*) error "File $1 can not be uncompressed with smartzip";;
esac
Here you notice that we use a new special variable called $1. This variable contains the first argument given to a program. Say we run
smartzip articles.zip
then $1 will contain the string articles.zip
The select statement is a bash specific extension and is very good for interactive use. The user can select a choice from a list of different values:
select var in ... ; doHere is an example:
break
done
.... now $var can be used ....
#!/bin/shHere is what the script does:
echo "What is your favourite OS?"
select var in "Linux" "Gnu Hurd" "Free BSD" "Other"; do
break
done
echo "You have selected $var"
What is your favourite OS?In the shell you have the following loop statements available:
1) Linux
2) Gnu Hurd
3) Free BSD
4) Other
#? 1
You have selected Linux
while ...; doThe while-loop will run while the expression that we test for is true. The keyword "break" can be used to leave the loop at any point in time. With the keyword "continue" the loop continues with the next iteration and skips the rest of the loop body.
....
done
The for-loop takes a list of strings (strings separated by space) and assigns them to a variable:
for var in ....; doThe following will e.g. print the letters A to C on the screen:
....
done
#!/bin/shA more useful example script, called showrpm, prints a summary of the content of a number of RPM-packages:
for var in A B C ; do
echo "var is $var"
done
#!/bin/shAbove you can see the next special variable, $* which contains all the command line arguments. If you run
# list a content summary of a number of RPM packages
# USAGE: showrpm rpmfile1 rpmfile2 ...
# EXAMPLE: showrpm /cdrom/RedHat/RPMS/*.rpm
for rpmpackage in $*; do
if [ -r "$rpmpackage" ];then
echo "=============== $rpmpackage =============="
rpm -qi -p $rpmpackage
else
echo "ERROR: cannot read file $rpmpackage"
fi
done
showrpm openssh.rpm w3m.rpm webgrep.rpm
then $* contains the 3 strings openssh.rpm, w3m.rpm and webgrep.rpm.
The GNU bash knows until-loops as well but generally while and for loops are sufficient.
Quoting
Before passing any arguments to a program the shell tries to expand wildcards and variables. To expand means that the wildcard (e.g. *) is replaced by the appropriate file names or that a variable is replaced by its value. To change this behaviour you can use quotes: Let's say we have a number of files in the current directory. Two of them are jpg-files, mail.jpg and tux.jpg.
#!/bin/shThis will print "mail.jpg tux.jpg".
echo *.jpg
Quotes (single and double) will prevent this wildcard expansion:
#!/bin/shThis will print "*.jpg" twice.
echo "*.jpg"
echo '*.jpg'
Single quotes are most strict. They prevent even variable expansion. Double quotes prevent wildcard expansion but allow variable expansion:
#!/bin/shThis will print:
echo $SHELL
echo "$SHELL"
echo '$SHELL'
/bin/bashFinally there is the possibility to take the special meaning of any single character away by preceeding it with a backslash:
/bin/bash
$SHELL
echo \*.jpgThis will print:
echo \$SHELL
*.jpgHere documents
$SHELL
Here documents are a nice way to send several lines of text to a command. It is quite useful to write a help text in a script without having to put echo in front of each line. A "Here document" starts with << followed by some string that must also appear at the end of the here document. Here is an example script, called ren, that renames multiple files and uses a here document for its help text:
#!/bin/shThis is the most complex script so far. Let's discuss it a little bit. The first if-statement tests if we have provided at least 3 command line parameters. (The special variable $# contains the number of arguments.) If not, the help text is sent to the command cat which in turn sends it to the screen. After printing the help text we exit the program. If there are 3 or more arguments we assign the first argument to the variable OLD and the second to the variable NEW. Next we shift the command line parameters twice to get the third argument into the first position of $*. With $* we enter the for loop. Each of the arguments in $* is now assigned one by one to the variable $file. Here we first test that the file really exists and then we construct the new file name by using find and replace with sed. The backticks are used to assign the result to the variable newfile. Now we have all we need: The old file name and the new one. This is then used with the command mv to rename the files.
# we have less than 3 arguments. Print the help text:
if [ $# -lt 3 ] ; then
cat <<HELP
ren -- renames a number of files using sed regular expressions
USAGE: ren 'regexp' 'replacement' files...
EXAMPLE: rename all *.HTM files in *.html:
ren 'HTM$' 'html' *.HTM
HELP
exit 0
fi
OLD="$1"
NEW="$2"
# The shift command removes one argument from the list of
# command line arguments.
shift
shift
# $* contains now all the files:
for file in $*; do
if [ -f "$file" ] ; then
newfile=`echo "$file" | sed "s/${OLD}/${NEW}/g"`
if [ -f "$newfile" ]; then
echo "ERROR: $newfile exists already"
else
echo "renaming $file to $newfile ..."
mv "$file" "$newfile"
fi
fi
done
Functions
As soon as you have a more complex program you will find that you use the same code in several places and also find it helpful to give it some structure. A function looks like this:
functionname()You need to "declare" functions at the beginning of the script before you use them.
{
# inside the body $1 is the first argument given to the function
# $2 the second ...
body
}
Here is a script called xtitlebar which you can use to change the name of a terminal window. If you have several of them open it is easier to find them. The script sends an escape sequence which is interpreted by the terminal and causes it to change the name in the titlebar. The script uses a function called help. As you can see the function is defined once and then used twice:
#!/bin/shIt's a good habit to always have extensive help inside the scripts. This makes it possible for others (and you) to use and understand the script.
# vim: set sw=4 ts=4 et:
help()
{
cat <<HELP
xtitlebar -- change the name of an xterm, gnome-terminal or kde konsole
USAGE: xtitlebar [-h] "string_for_titelbar"
OPTIONS: -h help text
EXAMPLE: xtitlebar "cvs"
HELP
exit 0
}
# in case of error or if -h is given we call the function help:
[ -z "$1" ] && help
[ "$1" = "-h" ] && help
# send the escape sequence to change the xterm titelbar:
echo -e "\033]0;$1\007"
#
Command line arguments
We have seen that $* and $1, $2 ... $9 contain the arguments that the user specified on the command line (The strings written behind the program name). So far we had only very few or rather simple command line syntax (a couple of mandatory arguments and the option -h for help). But soon you will discover that you need some kind of parser for more complex programs where you define your own options. The convention is that all optional parameters are preceeded by a minus sign and must come before any other arguments (such as e.g file names).
There are many possibilities to implement a parser. The following while loop combined with a case statement is a very good solution for a generic parser:
#!/bin/shTry it out! You can run it e.g with:
help()
{
cat <<HELP
This is a generic command line parser demo.
USAGE EXAMPLE: cmdparser -l hello -f -- -somefile1 somefile2
HELP
exit 0
}
while [ -n "$1" ]; do
case $1 in
-h) help;shift 1;; # function help is called
-f) opt_f=1;shift 1;; # variable opt_f is set
-l) opt_l=$2;shift 2;; # -l takes an argument -> shift by 2
--) shift;break;; # end of options
-*) echo "error: no such option $1. -h for help";exit 1;;
*) break;;
esac
done
echo "opt_f is $opt_f"
echo "opt_l is $opt_l"
echo "first arg is $1"
echo "2nd arg is $2"
cmdparser -l hello -f -- -somefile1 somefile2It produces
opt_f is 1How does it work? Basically it loops through all arguments and matches them against the case statement. If it finds a matching one it sets a variable and shifts the command line by one. The unix convention is that options (things starting with a minus) must come first. You may indicate that this is the end of option by writing two minus signs (--). You need it e.g with grep to search for a string starting with a minus sign:
opt_l is hello
first arg is -somefile1
2nd arg is somefile2
Search for -xx- in file f.txt:Our option parser can handle the -- too as you can see in the listing above.
grep -- -xx- f.txt
Examples
A general purpose sceletonNow we have discussed almost all components that you need to write a script. All good scripts should have help and you can as well have our generic option parser even if the script has just one option. Therefore it is a good idea to have a dummy script, called framework.sh, which you can use as a framework for other scripts. If you want to write a new script you just make a copy:
cp framework.sh myscriptand then insert the actual functionality into "myscript".
Let's now look at two more examples:
A binary to decimal number converter
The script b2d converts a binary number (e.g 1101) into its decimal equivalent. It is an example that shows that you can do simple mathematics with expr:
#!/bin/shThe algorithm used in this script takes the decimal weight (1,2,4,8,16,..) of each digit starting from the right most digit and adds it to the sum if the digit is a 1. Thus "10" is:
# vim: set sw=4 ts=4 et:
help()
{
cat <<HELP
b2h -- convert binary to decimal
USAGE: b2h [-h] binarynum
OPTIONS: -h help text
EXAMPLE: b2h 111010
will return 58
HELP
exit 0
}
error()
{
# print an error and exit
echo "$1"
exit 1
}
lastchar()
{
# return the last character of a string in $rval
if [ -z "$1" ]; then
# empty string
rval=""
return
fi
# wc puts some space behind the output this is why we need sed:
numofchar=`echo -n "$1" | wc -c | sed 's/ //g' `
# now cut out the last char
rval=`echo -n "$1" | cut -b $numofchar`
}
chop()
{
# remove the last character in string and return it in $rval
if [ -z "$1" ]; then
# empty string
rval=""
return
fi
# wc puts some space behind the output this is why we need sed:
numofchar=`echo -n "$1" | wc -c | sed 's/ //g' `
if [ "$numofchar" = "1" ]; then
# only one char in string
rval=""
return
fi
numofcharminus1=`expr $numofchar "-" 1`
# now cut all but the last char:
rval=`echo -n "$1" | cut -b 0-${numofcharminus1}`
}
while [ -n "$1" ]; do
case $1 in
-h) help;shift 1;; # function help is called
--) shift;break;; # end of options
-*) error "error: no such option $1. -h for help";;
*) break;;
esac
done
# The main program
sum=0
weight=1
# one arg must be given:
[ -z "$1" ] && help
binnum="$1"
binnumorig="$1"
while [ -n "$binnum" ]; do
lastchar "$binnum"
if [ "$rval" = "1" ]; then
sum=`expr "$weight" "+" "$sum"`
fi
# remove the last position in $binnum
chop "$binnum"
binnum="$rval"
weight=`expr "$weight" "*" 2`
done
echo "binary $binnumorig is decimal $sum"
#
0 * 1 + 1 * 2 = 2
To get the digits from the string we use the function lastchar. This uses wc -c to count the number of characters in the string and then cut to cut out the last character. The chop function has the same logic but removes the last character, that is it cuts out everything from the beginning to the character before the last one.
A file rotation program
Perhaps you are one of those who save all outgoing mail to a file. After a couple of months this file becomes rather big and it makes the access slow if you load it into your mail program. The following script rotatefile can help you. It renames the mailfolder, let's call it outmail, to outmail.1 if there was already an outmail.1 then it becomes outmail.2 etc...
#!/bin/shHow does the program work? After checking that the user provided a filename we go into a for loop counting from 9 to 1. File 9 is now renamed to 10, file 8 to 9 and so on. After the loop we rename the original file to 1 and create an empty file with the name of the original file.
# vim: set sw=4 ts=4 et:
ver="0.1"
help()
{
cat <<HELP
rotatefile -- rotate the file name
USAGE: rotatefile [-h] filename
OPTIONS: -h help text
EXAMPLE: rotatefile out
This will e.g rename out.2 to out.3, out.1 to out.2, out to out.1
and create an empty out-file
The max number is 10
version $ver
HELP
exit 0
}
error()
{
echo "$1"
exit 1
}
while [ -n "$1" ]; do
case $1 in
-h) help;shift 1;;
--) break;;
-*) echo "error: no such option $1. -h for help";exit 1;;
*) break;;
esac
done
# input check:
if [ -z "$1" ] ; then
error "ERROR: you must specify a file, use -h for help"
fi
filen="$1"
# rename any .1 , .2 etc file:
for n in 9 8 7 6 5 4 3 2 1; do
if [ -f "$filen.$n" ]; then
p=`expr $n + 1`
echo "mv $filen.$n $filen.$p"
mv $filen.$n $filen.$p
fi
done
# rename the original file:
if [ -f "$filen" ]; then
echo "mv $filen $filen.1"
mv $filen $filen.1
fi
echo touch $filen
touch $filen
Debugging
The most simple debugging help is of course the command echo. You can use it to print specific variables around the place where you suspect the mistake. This is probably what most shell programmers use 80% of the time to track down a mistake. The advantage of a shell script is that it does not require any re-compilation and inserting an "echo" statement is done very quickly.The shell has a real debug mode as well. If there is a mistake in your script "strangescript" then you can debug it like this:
sh -x strangescriptThis will execute the script and show all the statements that get executed with the variables and wildcards already expanded.
The shell also has a mode to check for syntax errors without actually executing the program. To use this run:
sh -n your_scriptIf this returns nothing then your program is free of syntax errors.
We hope you will now start writing your own shell scripts. Have fun!
References
- The man page and the info page of bash are very good and you will discover many more tricks there:
man bash
tkinfo bash
(I hate the standard info browser but tkinfo or konqueror (the url is info:/bash/Top ) are quite good) - The LinuxFocus article on GNU file utilities
- The LinuxFocus article on Regular expressions
- The LinuxFocus article on AWK
- The LinuxFocus article on Basic UNIX commands
- If bash isn't installed on your system or you want to get the newest version you can download it from the GNU website at http://www.gnu.org (to find out if bash is already installed on your system just type bash. If you don't get an error message bash is already installed)
Talkback form for this article
Every article has its own talkback page. On this page you can submit a comment or look at comments from other readers:
订阅：
博文 (Atom)
