What database does Google use?

I was wondering what database system does google use. Oracle or MySQL? These two are the names came into my mind then. But just after little googling I found something much more interesting.

Google actually does not use any of the database systems available at market. Sure they use Oracle and MySQl (modified versions) for some of their services but neither of those contributes to their Search Engine. They have developed a storage system for their search engine(really a google job!!!). This is called Big Table. Google has been using it since early 2005.

This is a distributed storage system for managing structured data that is designed to scale to a very large size: petabytes of data across thousands of commodity servers.

Bigtable is the basis of Google's search technology, as well as many other applications such Google Finance, Google Maps and Google Earth. These applications place very different demands on Bigtable, both in terms of data size (from URLs to web pages to satellite imagery) and latency requirements (from backend bulk processing to real-time data serving).

Bigtable was developed with very high speed, flexibility and extremely high scalability in mind. A Bigtable database can be petabytes in size and span thousands of distributed servers.

Google released it in May 2008 as part of Google App Engine. As is typical with Google offerings, Bigtable is free to use, and the service is described as "in beta," even though Google has been using Bigtable internally for more than three years.

So why not dig deeper to find more about this....let me know if any of you get lucky with something interesting.

MySQL :: Compiling the skeleton engine

From MySQL-5.1 there is support for pluggable storage engine. Here we will learn how to compile and use a skeleton engine.
First you need to have MySQL installed in your system. For installing follow this link

Now get the latest version of the skeleton engine. You will get it here
http://hg.tangent.org/skeleton-mysql-engine?ca=tip;type=gz

Now Step 1 untar and uncompress the file.

Now cd to this directory. You will see there is no configure file. Step 2 to generate on do the following

[root@localhost skeleton-mysql-engine-0beed5ef2859]# sh config/bootstrap+ '[' -f /usr/bin/glibtoolize ']'
+ LIBTOOLIZE=libtoolize
+ aclocal
+ autoheader
+ libtoolize --automake --force
+ automake --add-missing --force
configure.in:5: installing `config/missing'
configure.in:5: installing `config/install-sh'
src/Makefile.am: installing `config/depcomp'
Makefile.am: installing `./INSTALL'
+ autoconf

Now Step 3 is to configure

[root@localhost skeleton-mysql-engine-0beed5ef2859]# ./configure --with-mysql=/root/Desktop/mysql-5.1.30/ --libdir=/usr/local/mysql-5.1.30/lib/mysql/

Here the --with-mysql should be pointed to a source tree for MySQL. The --libdir can be used to say which lib directory the plugin should be installed in once it is compiled.

Now Step 4 is to just make, make install.

[root@localhost skeleton-mysql-engine-0beed5ef2859]# make
[root@localhost skeleton-mysql-engine-0beed5ef2859]# make install

Now start mysql and connect to it. Then Step 5 is to load it.

mysql> INSTALL PLUGIN skeleton SONAME 'libskeleton_engine.so';
Query OK, 0 rows affected (0.01 sec)

You can see all the storage engines.

mysql> show plugins;
+------------+--------+----------------+-----------------------+---------+
| Name | Status | Type | Library | License |
+------------+--------+----------------+-----------------------+---------+
| binlog | ACTIVE | STORAGE ENGINE | NULL | GPL |
| CSV | ACTIVE | STORAGE ENGINE | NULL | GPL |
| MEMORY | ACTIVE | STORAGE ENGINE | NULL | GPL |
| MyISAM | ACTIVE | STORAGE ENGINE | NULL | GPL |
| MRG_MYISAM | ACTIVE | STORAGE ENGINE | NULL | GPL |
| SKELETON | ACTIVE | STORAGE ENGINE | libskeleton_engine.so | BSD |
+------------+--------+----------------+-----------------------+---------+
6 rows in set (0.00 sec)

See,you have loaded your first storage engine.

MySQL :: Installing in Linux from source

What should be the Step 1 for installing MySQL from source. Easy,download the source from MySQL site. Here is the link MySQL Source

Here is a point, If you are going to work with storage engine download at least mysql-5.1 because older versions don't have pluggable storage engine support. In this example I am using mysql-5.1.30 in Fedora 8.
It may take a few minutes to download the source. By the time you get the source downloaded you can read the rest of the article or take a cup of tea or just keep starring at the monitor(that's what I do actually).

Now make sure that you are logged in as superuser(root). Now create an account mysql. Step 2

# groupadd mysql
# useradd -g mysql mysql

This account(mysql) will have access to manage databases.

Now unpack the downloaded file and cd to it. Step 3

# tar -xzf mysql-5.1.30.tar.gz
# cd mysql-5.1.30

Now configure, make, make install. I usually use the /usr/local/ directory to install mysql.
Step 4

# ./configure --prefix=/usr/local/mysql-5.1.30
# make
# make install

Now we have to create a configuration file. It offers several security and control options (here you can limit system resources to be used by MySQL server, set the default collation and character set etc.). No need to make a new configuration file,there are 4 in support-files/ directory. Here we will use my-small.cnf. Step 5

# cp support-files/my-small.cnf /etc/my.cnf
# chown root /etc/my.cnf
# chgrp root /etc/my.cnf
# chmod 644 /etc/my.cnf

Now open the file and add user = mysql below [mysqld]

Now we have to install the grant tables. Installing the grant tables is accomplished by running the mysql_install_db script. This creates the mysql database which will hold all database privileges, the test database which you can use to test MySQL, and also privilege entries for the user that run mysql_install_db and a root user (without any passwords).Step 6

# /usr/local/mysql-5.1.30/bin/mysql_install_db --user=mysql

The script will create /usr/local/mysql/var/ directory containing the necessary databases. This directory serves as a default storage for all databases you will create. Make sure it is writable by "mysql" system user!

Now we shall start the server. Step 7

# /usr/local/mysql-5.1.30/bin/mysqld_safe --user=mysql &

This will start the server. To check if really server is running you can check the processes or try checking the mysql version.

# ps -all|grep mysql
# /usr/local/mysql-5.1.30/bin/mysqladmin version

Now connect to mysql server. Step 8

# /usr/local/mysql-5.1.30/bin/mysql -u root

Now you can see that you are connected to mysql. To check all Hosts and users do,

mysql> SELECT Host, User FROM mysql.user;

Right now there is no password for root user,you can put a password for the root user.

mysql> SET PASSWORD FOR 'root'@'localhost' = PASSWORD('123456');
Query OK, 0 rows affected (0.00 sec)

To exit mysql write

mysql> quit

Now to stop the server,

# /usr/local/mysql-5.1.30/bin/mysqladmin -u root -p shutdown

Now to start it again,

# /usr/local/mysql-5.1.30/bin/mysqld_safe --user=mysql &

Now connect again,

# /usr/local/mysql-5.1.30/bin/mysql -u root -p

You will be now prompted for password and giving the password you are connected to mysql.

If everything is done properly than Congratulations, mysql installation from source is done in your PC.

Appendix:
One more thing, If you don't want to start MySQL server manually after each system reboot set up an automatic startup. Go back to the directory where you extracted the downloaded mysql tarball file. Enter
# cp support-files/mysql.server /etc/init.d/mysql
# chmod 755 /etc/init.d/mysql
# chkconfig --add mysql
# chkconfig --level 35 mysql on

MySQL :: Step by step hacking into MySQL

First of all a quiz.Do you know which database has been used in Facebook?
Guess guess guess.......

its MySQL. This is one of the biggest MySQL implementation in the world. So I think you should have some more respect for MySQL now. If you want to learn more about MySQL than my blog might help you. Here I will discuss on how to make the best use of this open source software,specially on storage engine like how to make your own storage engine etc. Here are the links to my articles on MySQL.

1. MySQL :: Installing in Linux from source

2. MySQL :: Uninstalling


3. MySQL :: Compiling the skeleton engine