I have a java program that crawls for some data on some sites and inserts it into the database.
The Program keeps doing this :
- Get the html
- Extract the relevant data with some splits
- Insert into to database
For the first 5-10 min it runs perfectly and very fast but after 20-30 min it becomes too slow and suddenly the program takes 80% cpu (in the beginning it takes something like 4%).
Do you know any reason why this might happen?