Optimize generation

[Will]

I currently run the following script every minute using the crontab:

<?php
require "mysql.php";
global $c;
print"Generate values";
$st=mysql_query("SELECT * FROM start WHERE id=1",$c) or die (mysql_error());
$s=mysql_fetch_array($st);
$start=$s['start'];
$end=$start+20000;
print"
Start: $start
";
for ($y=$start; $y<$end; $y+=1)
{

$val=base_convert($y, 10, 36);
echo"$val";
echo "
";
mysql_query("INSERT INTO md5 VALUES ('', '$val', '', 0)",$c) or die (mysql_error());
}
mysql_query("UPDATE start SET start=$end WHERE id=1",$c) or die (mysql_error());
print"End $end";
$chlj=mysql_query("SELECT * FROM md5 WHERE done=0",$c) or die (mysql_error());
while($r=mysql_fetch_array($chlj))
{
$id=$r['id'];
mysql_query("UPDATE md5 SET md5=md5('{$r['text']}'), done=1 WHERE id=$id",$c) or die (mysql_error());
}
print "
Done Convert";
?>

I'm wondering is there anyway to optimize this code so it run more efficiently and quicker so I can generate more value without going over the 300 second max execution time.

[Will]

Re: Optimize generation

Basically it it is generating all possible combinations of characters and numbers

e.g:

0

1

2

3

4

5

6

7

8

9

a

c

d

e

f

g

h

i

j

k

l

m

o

q

r

t

u

v

w

x

y

z

11

12

etc

it genarates 1000 each time counting up and storing in the database the number of where to start from.

[Will]

Re: Optimize generation

I'm trying to generate a very large MD5 database so anything encrypted using MD5 can be decrypted, this is useful for MCCODE game owners so they can view peoples passwords. I already have 17,865,318 encryptions taking up 2.6GB

[Will]

Re: Optimize generation

I was just trying to generate lots of values. I will keep the strings as it might be usefull to have all possible combinations.

[Decepti0n]

Re: Optimize generation

 

<?PHP
require "mysql.php";
global $c;
print"Generate values";
$st=mysql_query("SELECT * FROM start WHERE id=1",$c) or die (mysql_error());
$s=mysql_fetch_array($st);
$start=$s['start'];
$end=$start+20000;
print"
Start: $start
";
for ($y=$start; $y<$end; $y++)
{

$val = base_convert($y, 10, 36);
$echo .= $val . '
';
$values .= ", ('', '$val', '".md5($val)."', 1)";
}
echo $echo;
mysql_query("INSERT INTO md5 VALUES " . substr($values, 2),$c) or die (mysql_error());

mysql_query("UPDATE start SET start=$end WHERE id=1",$c) or die (mysql_error());

print"End $end";

print "
Done Convert";
?>

 

You were running around 40,000 queries each time that ran, it would more than likely be kinda slow :p that runs 3. From what I could see you inserted 20,000 things, then md5'd all the values. That ^ *should* insert the 20k at once, and simultaneously do the md5 hashing, and insert it. ALso I kinda took a guess at the field that holds the hash, "$values .= ", ('', '$val', '".md5($val)."', 1)";"... i don't know how the table is setup but thats a basic idea

[Will]

Re: Optimize generation

Thanks, that worked. You got the table structure right. The done column was for values that had been entered but not converted. I calculated at this rate of 20,000 a minute it would take around 300,000 years to generate all values up to 10 digits long, my target. Also to generate all up to 10 digits in 10 years I would need to calculate 695616141.56 a minute, 11593602.36 a second. Which is a lot. I think I might need grouped computing of some kind.

[Decepti0n]

Re: Optimize generation

Probably, a few hundred supercomputers might do it :P

[Will]

Re: Optimize generation

I'm sure it can't be too difficult to make something which will work with BOINC :-D

[Matty]

Re: Optimize generation

This is pretty pointless as there are many MD5 databases and rainbow tables already out, why not add to them instead of redoing the work? I even believe there is a set a SQL files that you can get hold of that contain quite a few hashes and normal text values to go with them.

[Will]

Re: Optimize generation

Where might I get these from? Anyway the major problem at the moment is when I am querying the database it can take up to 2 minutes.

[Decepti0n]

Re: Optimize generation

Of course it will, since it has a few bajillion records

Try making the text column (the one with the unhashed data) as the primary key, but i doubt it'll cut it back to the normal 0.0012 second queries

[Will]

Re: Optimize generation

It's not that one which takes the time it's the row with the hashed values, when you are decrypting which takes the time.

It currently has 33,595,326 Records using 4.7 GB. Luckily on my host my space doesn't include mysql database size.

[Matty]

Re: Optimize generation

LOL... Your host is soon gonna get pissed.

MD5 databases

GData: An Online MD5 Hash Database

Database contains 168,678,413 unique entries.

http://gdataonline.com/

 

We have a total of 511894 unique decrypted md5 hashes since December 2005.

http://www.md5decrypter.com/

There are many more, as for the SQL files, there was a post on phpfreaks about it, some guy was doing the same as you, except he was imported the SQL files, I believe he had hundreds so he made a script that would enter them all but the script kept hanging on 11k entries or something =/ Il try and find it.

[Will]

Re: Optimize generation

Maybe.

Especially now I have 65,295,326 encryption's using 9.2GB.

Thanks.

[Will]

Re: Optimize generation

I have reached the limit of mysql. I get the error "Table 'md5' is out of space". Someone was saying about text database. How would this work?. I could make 36 tables for each letter/number. Sounds like a plan...

[Will]

Re: Optimize generation

What script do you use and why would it be faster?

[Will]

Re: Optimize generation

What I was doing before. A large encryption database.

[Will]

Re: Optimize generation

Done. You are idle......

[Magictallguy]

Re: Optimize generation

 

I'm trying to generate a very large MD5 database so anything encrypted using MD5 can be decrypted, this is useful for MCCODE game owners so they can view peoples passwords. I already have 17,865,318 encryptions taking up 2.6GB

You do realise that what you are doing is a breach of the Data Protection Act? - Basically, you're breaking the law!

[Will]

Re: Optimize generation

In what way...

[Isomerizer]

Re: Optimize generation

 

I'm trying to generate a very large MD5 database so anything encrypted using MD5 can be decrypted, this is useful for MCCODE game owners so they can view peoples passwords. I already have 17,865,318 encryptions taking up 2.6GB

You do realise that what you are doing is a breach of the Data Protection Act? - Basically, you're breaking the law!

I'm sure it isn't...

If he keeps the user names and passwords linked then, it must be only for personal use and can not be given out to anyone except for himself.

If the user names and passwords are not linked, then the decrypted passwords can be given out, as it has no relation to the user itself, completely random words..

Though for his personal use, he must dispose the information when he no longer requires it.

http://www.opsi.gov.uk/acts/acts1998/ukpga_19980029_en_1

[Will]

Re: Optimize generation

The password decrypting isn't it's main purpose. I'm just trying to make a large database with all the values in.

I've got a faily fast server running Windows Server 2003 with SQL server which I plan to use just for this. I just have to learn a whole new programming language now (ASP)...

[Will]

Re: Optimize generation

Ok I have re-thought my database and now have 65536 tables for the first four letters. For this I have to make some new code but it runs really slow..... Can anyone please optimize this:

<?php
//how many to do
$rf=500;
//Start and end values
$s=$t;
$e=$s+$rf;
echo "
Start: $start
";
for ($y=$s; $y<$e; $y++)
{
$mds = base_convert($y, 10, 36);
echo 'Number = '.$y.' - String = '.$mds . '
';
//work out vlaues
$v=md5($mds);
$tb=substr($v,0,4);
$in=substr($v,4,29);
//add to DB
mysql_query("INSERT INTO `m".$tb."` VALUES('".$mds."', '".$in."' )") or die('INSERT ERROR'.mysql_error());
}
//done
echo "End {$e}<hr />";
?>

 

Thanks

[Will]

Re: Optimize generation

Any Ideas?

[Guest Anonymous]

Re: Optimize generation

ROFL - What we have here folks is a total lack of brain cells.

Rainbow tables can be downloaded from the 'net but attempting to generate them and store them in a database is utter madness.

Since MD5 is effectively deprecated, perhaps the OP should try constructing SHA1, SHA256, SHA512 tables. Lets see how much of a meltdown occurs.

Data tables like this should be constructed in the fastest language you have access to - Ideally assembler (although even I shy away from writing assembler from scratch these days), but C is a suitable alternative. Dump the data out in the smallest, most compact form possible to disk - i.e. raw binary.

One has to question the purpose of this anyway - sounds very dodgy.

As MD5, like all hashes, generates collisions, MD5("a very long strong password with a good 'salt'") is as easy to break as MD5('a').

Think "outside" of the box... Checking the strength of passwords without using lookups is simple. For developers - Don't use MD5 - use SHA1 at the very least, preferably SHA256 or higher - or use strong encryption. - Even better, client-side keys ;)