Take the 2-minute tour ×  I have a MySql table which consists
of:    1. ~25million rows (CURRENTLY)   2. 3 indexes   3. Each
day, a crawler adds ~3million rows   4. I'm currently not
looking too far, but a final estimate of the db can be ~CONST*e9
rows   5. Currently 9.5giga   6. innodb and it is being read
from while inserting  The data itself consists of a text of ~100
chars + several fields with meta data about it. The indexes are
the unique id, the writer name, and the writer id.  Till now,
everything went smooth, but now the server is having a hard time
handling the inserts of the new data (~10seconds for each insert
which adds ~3k lines). I'm trying to find ways to overcome this
issue. Things I consider:    1. Doing the index while inserting
takes effort. Maybe not doing it while inserting, and only after
X inserts adding the indexes.   2. Partitioning the data into
different tables.   3. Crawling into a small db, and each X
minutes/days, moving the data into the big db.   4. Moving to a
different db. I