Comments for isnerd.net http://isnerd.net Systems Administration, IT Management Sat, 22 Nov 2008 03:12:38 +0000 http://wordpress.org/?v=2.6.1 Comment on Faster Than Grep by S http://isnerd.net/2008/02/26/faster-than-grep/#comment-260 S Thu, 04 Sep 2008 03:41:44 +0000 http://isnerd.net/2008/02/26/faster-than-grep/#comment-260 This sounds a lot like my job. We don't usually get data sets *that* big, but I'm not gonna bet against it, either. This sounds a lot like my job. We don’t usually get data sets *that* big, but I’m not gonna bet against it, either.

]]> Comment on The Ultimate Systems Administrator (So you want to be a Systems Administrator? Part 4) by James Hicks http://isnerd.net/2007/11/03/the-ultimate-systems-administrator-so-you-want-to-be-a-systems-administrator-part-4/#comment-166 James Hicks Mon, 21 Jul 2008 10:07:18 +0000 http://isnerd.net/2007/11/03/the-ultimate-systems-administrator-so-you-want-to-be-a-systems-administrator-part-4/#comment-166 Congrats John :) Congrats John :)

]]> Comment on The Ultimate Systems Administrator (So you want to be a Systems Administrator? Part 4) by John http://isnerd.net/2007/11/03/the-ultimate-systems-administrator-so-you-want-to-be-a-systems-administrator-part-4/#comment-163 John Sun, 20 Jul 2008 13:42:08 +0000 http://isnerd.net/2007/11/03/the-ultimate-systems-administrator-so-you-want-to-be-a-systems-administrator-part-4/#comment-163 Hey James this is John again from 2 comments up. Just wanted to let everyone know I finally made it to SysAdmin! I took upon myself of learning a lot of bash scripting (with sed/awk mixed in) and moved to a big city. I was given my chance by a big insurance company as a Jr Linux/Backup SysAdmin. Don't give up on your dream peoples, it is possible! Hey James this is John again from 2 comments up. Just wanted to let everyone know I finally made it to SysAdmin! I took upon myself of learning a lot of bash scripting (with sed/awk mixed in) and moved to a big city. I was given my chance by a big insurance company as a Jr Linux/Backup SysAdmin. Don’t give up on your dream peoples, it is possible!

]]> Comment on 3Ware 24 Disk 600MB/sec RAID6: An affordable mini-SAN? by ecards http://isnerd.net/2007/11/01/3ware-24-disk-600mbsec-raid6-an-affordable-mini-san/#comment-135 ecards Sun, 22 Jun 2008 17:31:26 +0000 http://isnerd.net/2007/11/01/3ware-24-disk-600mbsec-raid6-an-affordable-mini-san/#comment-135 man I would love to see a test of the 3ware in that config to see if they really perform as advertised. will probably take the plunge anyway for a smaller setup. man I would love to see a test of the 3ware in that config to see if they really perform as advertised.

will probably take the plunge anyway for a smaller setup.

]]> Comment on Faster Than Grep by James Hicks http://isnerd.net/2008/02/26/faster-than-grep/#comment-115 James Hicks Thu, 10 Apr 2008 10:50:35 +0000 http://isnerd.net/2008/02/26/faster-than-grep/#comment-115 egrep: I'm guessing egrep wont accept a 180KB + overhead commandline. It was probably tried before they tried looping/nesting greps, which was before they handed me the problem. However, if you could do that, I'm sure it would work. It might take many hours to chew through it all rather than a few hours... but it would work. egrep: I’m guessing egrep wont accept a 180KB + overhead commandline. It was probably tried before they tried looping/nesting greps, which was before they handed me the problem.

However, if you could do that, I’m sure it would work. It might take many hours to chew through it all rather than a few hours… but it would work.

]]> Comment on Faster Than Grep by egrep http://isnerd.net/2008/02/26/faster-than-grep/#comment-114 egrep Thu, 10 Apr 2008 00:43:09 +0000 http://isnerd.net/2008/02/26/faster-than-grep/#comment-114 egrep '^(opt0|opt1|opt2|...|opt219)' file maybe I missed something, but why would this not work? egrep ‘^(opt0|opt1|opt2|…|opt219)’ file

maybe I missed something, but why would this not work?

]]> Comment on Faster Than Grep by James Hicks http://isnerd.net/2008/02/26/faster-than-grep/#comment-113 James Hicks Wed, 09 Apr 2008 09:44:22 +0000 http://isnerd.net/2008/02/26/faster-than-grep/#comment-113 Jesus: Given that we only needed to scan 13k records to find the unique members of the first field, any approach that involved a computer was likely to work :) Geoffrey: I don't claim to be a proficient C coder either ;) Iain: My C sources are now the property of my previous employer, who probably have not kept them. Apolodor: There were not 220 unique fields in the 40GB of data... of the 13,000 we wanted, the first field had only 220 unique members. Jesus: Given that we only needed to scan 13k records to find the unique members of the first field, any approach that involved a computer was likely to work :)

Geoffrey: I don’t claim to be a proficient C coder either ;)

Iain: My C sources are now the property of my previous employer, who probably have not kept them.

Apolodor: There were not 220 unique fields in the 40GB of data… of the 13,000 we wanted, the first field had only 220 unique members.

]]> Comment on Faster Than Grep by Jesus DeLaTorre http://isnerd.net/2008/02/26/faster-than-grep/#comment-112 Jesus DeLaTorre Tue, 08 Apr 2008 18:56:27 +0000 http://isnerd.net/2008/02/26/faster-than-grep/#comment-112 How did you find the unique records in the first field? The naive approach can take o(n^2). One way is the have the fields added to a binary tree and then reduce it to an array. This probably would had increased the programming overhead though. Another approach is to sort the first field and pick the unique records this way. This would probably be the easiest approach for a 0(n lg n) approach. How did you find the unique records in the first field? The naive approach can take o(n^2). One way is the have the fields added to a binary tree and then reduce it to an array. This probably would had increased the programming overhead though. Another approach is to sort the first field and pick the unique records this way. This would probably be the easiest approach for a 0(n lg n) approach.

]]> Comment on Faster Than Grep by Geoffrey http://isnerd.net/2008/02/26/faster-than-grep/#comment-109 Geoffrey Mon, 07 Apr 2008 02:30:12 +0000 http://isnerd.net/2008/02/26/faster-than-grep/#comment-109 Good story, enjoyed reading. I have had similar stressful situations but not quite like yours - I am not as proficient as C coder as yourself, probably would have searched the web for something similar to what I wanted and hacked it to work then hoped for the best :) Good story, enjoyed reading.

I have had similar stressful situations but not quite like yours - I am not as proficient as C coder as yourself, probably would have searched the web for something similar to what I wanted and hacked it to work then hoped for the best :)

]]> Comment on Faster Than Grep by lain_proliant http://isnerd.net/2008/02/26/faster-than-grep/#comment-88 lain_proliant Thu, 20 Mar 2008 09:16:29 +0000 http://isnerd.net/2008/02/26/faster-than-grep/#comment-88 I personally love C, but I definitely agree that it is very hard to hack something up quickly and get it to work flawlessly. Juggling pointers and memory allocation in C is somewhat of an art form, and it takes a lot of care and practice, and most of all _time_. Languages like Perl and PHP are definitely more suited for quick ditties, but have subsequent hits in performance, which are usually not noticed when working on data several magnitudes smaller than your example. Its really just a function of whether or not you can afford the extra processing time. If this is a general solution that can be used on many large files in the future, you may want to keep those C sources. I personally love C, but I definitely agree that it is very hard to hack something up quickly and get it to work flawlessly. Juggling pointers and memory allocation in C is somewhat of an art form, and it takes a lot of care and practice, and most of all _time_. Languages like Perl and PHP are definitely more suited for quick ditties, but have subsequent hits in performance, which are usually not noticed when working on data several magnitudes smaller than your example. Its really just a function of whether or not you can afford the extra processing time. If this is a general solution that can be used on many large files in the future, you may want to keep those C sources.

]]>