Every once in a while I hear of someone losing their data because they neglected the fact that they need to perform backups on a regular basis. One of friends calls me up one day and tells me she lost 10 years worth of pictures when her hard drive crashed and was begging for help. Luckily I was able to recover her pictures and other data, but she was lucky, because I had the right tools for the job and the drive was physically accessible.
When a web server crashes, it is a completely different story, because you are at the mercy of the hosting provider and you do not have physical access to the drive(s). What most people do not realize is that their hosting provider does NOT backup their data. This is especially true of shared hosting…you know, the $10 per month type of accounts. Well, now you know and I hope you will do something about it. Data protection isn’t fun, but neither is paying for insurance. Consider your backups as insurance…it is a smart thing to do.
The best news is that is it very easy to automate this process. I even created a short video with detailed instructions on how to get this done for cPanel-based hosting accounts. The video is going to show you how to confiure the backup scripts on the server to perform daily backups, but that is just the first step. In order for your backups to do you any good, you have to store them somewhere other than your server. I show you how to do that as well. If you have any questions, leave a comment on this page or ask me on Twitter (http://twitter.com/WilsonMattos).
Resource Links
Auto FTP Manager: Download Auto FTP Manager
cPanel-based Web Hosting: Hostgator
Scripts
backup.sh (Note: everything goes in one line, just copy and paste from here):
Cron Job (Note: Do not remove the quotes):
Video Transcript
In this short video I’m going to show you a very simple way to automatically back up all the data on your web hosting server including your website files, MySQL databases, blogs, etc. So let’s go ahead and get started.
The very first step is to access the cPanel administration interface for your hosting account. I’m using HostGator as my hosting provider, and I’m using the default HostGator skin. The skin is the look and feel of your administration panel. Your interface may look a little bit different depending on what skin you’ve chosen or who your hosting provider is, but it’ll have the same features that I am demonstrating here, like all of these icons, for example.
Find the “Find” box (I like to use it so I don’t have to scroll through all the icons) and type the word “File” in it. Or just find the application called “File Manager.” Go ahead and click on “File Manager,” and then click on “Go.” When it comes up click the “New File” icon on the top left. And then type the name “backup.sh” Again, “backup.sh.” Then click the “Create New File” button. You will then be able to scroll down and find the “backup.sh” file you just created. Notice it is zero bytes. There’s nothing in it yet. Check the box right next to it, and then let’s click the “Edit” button on the top of the screen. Next, click the “Edit” button on the window that comes up above the text editor. Once the text editor is up, go ahead and copy the text that is underneath this video on this webpage in the box titled “backup.sh.” Go ahead and just copy all of it and paste it right here, and do not make any modifications whatsoever to it, then go ahead and click on “Save Changes.” Once that is done you can close the file manager window, and you can go ahead and close this window as well.
Now, let’s go back to our main cPanel window. The next step is to search for the “cron” application. There it is, “Cron Jobs.” Go ahead and click on it and select the “Standard” button to schedule a job. Now underneath this video there’s also a text box that is labeled “Cron Job.” Go ahead and copy the contents of that box. I’m going to just go ahead and type it here, and it’ll look like this. And make sure to change the last three fields of this command. The first one will be the user name, so my user name is this; my password, and then my e-mail address. So go ahead and make sure you put values for yourself here. The first couple of parts stay the same. Then you put in your user name for cpanel. This is your cPanel account, your cPanel password, and any e-mail address that you want the confirmation of your backup to be sent to. Then you pick the date and time that this is going to occur. So here you have the hour, so I have selected 3 AM. Let’s go ahead and change it to 2 AM. Actually, I want that to be 2:15 AM, so you select the hours and the minutes, so 2:15 AM. I want to run it every day of every weekday, every month. So the only changes here is the hours and minutes, and once I’m done I’m going to go ahead and “Save Crontab.” That’s it. I have now scheduled a job that will automatically back up my server on a daily basis.
So how do we get your backups? Okay, so we go back to the main interface of the HostGator cPanel, and we’re going to go ahead and find an application called backups. Here is the “Backups” application. I’ll go ahead and click on it. Then I will click on “Download or Generate a Full Backup.” There are the backups that have automatically run. These actually ran on November 18th, 2008, at 12:30am; 30 minutes past midnight November 20th; November 19th as well. So in order for me to download the backups, I just click on the backup and save the file locally. Now, as you can imagine this is another task for you to handle every day. Not only do you have to download these backups, but you would also have to go ahead and go to file manager and delete them after you back them up, because otherwise they’re going to just fill up the disk space and take up all the disk space on your hosting account. So let’s go ahead and automate this process as well.
The very first thing I’m going to do is I’m going to “My Documents,” and then I’m going to create a folder to store my backups. So let me call this “Web Server Backups.” You can call it whatever you like. We’re going to use this in the next step. And then I’m going to launch a little application called Auto FTP Manager. I’ve included a link to this application below this video. It’s a very cool little application, and they give you a free trial. So what I’m using to record this video is the free trial. And once you start the application a little wizard comes up. And we’re going to choose “Create Automated Transfer Profile.” It’s the second option, and I’m going to go ahead and click on “OK.” What I want is an “Automated Transfer.” I’ll go ahead and click on “Next.” And then I have to browse where I want to store those backups. So I just browse to the folder I created. Again this doesn’t matter. Call it whatever you’d like. Click on “OK.” And then I’ll choose “Next.” “FTP Server” is the correct choice on this screen, so I’ll just go ahead and click on “Next.” And then it wants me to type in the information for the FTP Server. So I’ll just type in my domain information, and my user name, and my password. Okay, and then I will select the FTP folder directory I want to back up. I’ll just say slash – Now this is the next very important step here under select sub-folders and files to include or exclude, go ahead and click on “Configure.” And the very first thing we want to do is click on “Deselect All.” This is important. You don’t want anything selected. Now, if you have backups that you have already created like I have, for example, like I showed you before. Then you want to check them to make sure that they get transferred, because otherwise they won’t get transferred the first time. If you haven’t created a backup yet, because your scheduled job on the server that we just created hasn’t run, then don’t worry about this. But I already have three backups, and I want them to be transferred. So I’ll go ahead and put check boxes right next to those. Click on “OK,” and the names of those backups were right here, so I knew what they were. Go ahead and click on “Next.” The default action here is not what we want. We want the transfer files from my domain, whatever your domain is called, to your machine. And then we want to select here the delete file from my domain after transfer. This automatically does the cleanup for you that I talked about, so that you’re not filling out your entire quota on your hosting account.
Okay, then we have to set a filter, so that we transfer the right file types. I’m going to go ahead and click on “Configure,” and I’m going to select “File Type.” I’m going to say “Include” rather than exclude, so I would like to “Include.” And then I’ll come here and type “.gz.” So I want to add files of type “.gz.” So it says now to include .gz. And this will always just grab those files for you. Then we’re going to go ahead and schedule this. We don’t want to schedule just once, so we want to say “Daily.” As you can see this is a pretty cool application. You can set up custom intervals; monthly, weekly, whatever you want. So I’m going say “Daily.” It’s 12:42, so let’s go ahead and make it the 12:43 schedule. It’s going to occur every “1″ day, and it is enabled. And then we’ll go ahead and click on “Next.” We’ll specify a name for this profile. Let me just call it HostGator, since this is what I’m backing up. You can call it again whatever you want. We don’t want to do anything after we’re finished. We’re just going to go ahead and “Finish” this, and as you can see here you have an automatic transfer profile. If you wanted to change a schedule or anything like that, you could right-click and edit that profile. Let’s take a look at the scheduled profile, and this tells us when it’s going to run. It’s about to run, so we just made that schedule; ten seconds. This should now transfer those files, delete them from our hosting account, so that everything is cleaned up. So here we go. It’s running, and it shows us what is going to happen. It’s scanning the files. Look, it found the files. It is transferring the files now. It is transferring all my backup files. And if we scroll here, file completed. Everything’s completed. Transfer completed and then file deleted. And we can go ahead and close this log. You can close this application. It doesn’t actually terminate the application. It stays running in your system tray. This is required in order for the job to run next time. And if we go to the scheduled profiles again, it says that in 23 hours, 59 minutes, and 20 seconds it’s going to run again. So we want to make sure that this application stays running, so that the scheduled jobs can run.
If we go back to the HostGator cPanel, we should see that those backups are no longer there. So we’ve cleaned up our home directory and removed the generated full backups. We no longer have a backup here. We’ve transferred them all. If I go to my documents and look at my web server backups, I can see that the files are there. This is my backup that I can use to restore my web server. That’s how simple that is. I hope you enjoyed the video and that this process serves you very well.
Hi will!
Thanks for the tutorial. I had some issues getting this going too – a whole bunch of copy and paste issues on my part, so thought I’d share with the group!
I needed to make sure I was using the correct – characters. When opening in a text editor I discovered half of them were the long hyphens, not the simple minus sign. Working a treat now, thanks again!
Hi Wil,
I copied and pasted the tags you have above and created my backup file.
The only problem is — Auto FTP Manager — found 2 *.gz files on my hostgator account, copied them, and then deleted them. These files were:
sitemap.xml.gz from my /public_html/ directory and
wp-tinymce.js from my /wp-includes/js/tinymce/ directory
LOL, yet it didn’t touch the backup*.gz file that is residing outside the /public_html/ directory.
How do I fix this? As I followed your instructions to the “T”.
Thanks for your help!
– Bonnie
Thank you for this excellent tutorial. I used this script to setup automatic backups for a site I manage, however, something has stopped working. The last automatic backup I received was November 16th. I use hostgator as well (shared hosting) and out of the blue, I just quit receiving backups. I checked with support & they say that Cron appears to be working fine, and they say the backup.sh looks fine (I didn’t change anything) So I’m wondering if something got changed by hostgator that would cause the backup.sh to become ineffective. I did try changing the permissions to 755 but that didn’t seem to help. I’d appreciate any advise you could give me to get this working again.
I tried this method on Hostgator and also could not get it to work. After some investigation, I found that the newer version of cPanel does not allow http authentication and therefore this method would not work since it relies on http authentication. Is there any workaround that uses POST to authenticate?
I tried using –keep-session-cookies and –save-cookies cookies.txt, but I’m not versed enough in Linux commands to make it work.
Hey Jane
Had the same problem.
- 2 of my Hostgator hosting accounts worked perfectly.
- 1 stopped working in about October
Finally got a chance to look at it last week. Just got this response from Hostgator Support.
——————-
I am convinced (wget believer here). I apologize for the delay. I have your script working again. For your script I had to add –auth-no-challenge like so:
wget -O /dev/null -q –auth-no-challenge –http-user=$1 –http-password=$2 http://localhost:2082/frontend/x3/backup/dofullbackup.html –post-data=”dest=homedir&email=$3″
The full backups appear to be running successfully. You may want to add that parameter to your others so that they don’t suddenly stop at some point. I don’t know what is different about this particular server. It could be some cPanel updates or something.
If you need anything else just let us know. We are here to help.
Best regards,
Richard B
Linux Systems Administrator
HostGator.com LLC
http://support.hostgator.com
——————————————-
Hope that helps.
Paul
Paul, thank you so much for this information. I had contacted Hostgator a couple times about this issue and they were never able to tell me anything other than, “the cron looks fine” & “the script looks fine”
One thing to note is the the text copied from the site here doesn’t work, it includes invalid asci characters. I noticed when trying to update the script that when I saved it it kept defaulting the encoding to “UTF-8″, I was finally able to get it to save as asci and now it’s working again.
Thank you for taking the time to post your solution!
We are using Crucial hosting, they use cpanel but they have turned off the full backup feature. Can you help by providing similar instructions on how to backup a home directory and the individual databases.
That would be alot of help!
Thanks
Joseph, you would have to write a script to do this. If you are looking to backup WordPress sites, then check out WP Twin, which makes it super simple.
Wilson,
This is amazing! I listened to your webinar on 7/7/10 hosted by James J Jones and purchased your Easy Redirect Script and have it configured on a new domain I purchased based upon your webinar advice.
Then I started poking around to see what else you have and found S3FlowShield and then Auto FTP Manager! Then I read the above comments and found some additional info on backing up to Amazon S3.
I am building websites for myself as well as customers and all of this is very useful. There is always the question of how to systematically do backups and it looks like I now have some good approaches to use.
– Thanks, Roger
I’ve ruin my website with jom sef … so the siteis having some problems…
I have backup my website exaclty like in this tutorial , i have the backup , if i ask the admin from hostgator to do a restore will hi be able to doi’t ( sorry for my english ) ? or what/how should i ask ? i need help pls …
If this help someone – it erase the archives older then 6 days …
find /home/username/ -name ‘backup-*’ -mtime +6 -exec rm {} \;
Flavius,
Yes, if you ask the admins at Hostgator, they should be able to restore the backup for you. Their helpdesk is really good!
Wil
Well this really IS a fantastic link, given to me by your friend May. I have been searching for an answer and it’s only thanks to dedicated people like this that people like us can breathe at night!
Funny really because some marketers rely on dozens if not hundreds of websites for their full time income, and at any point in time their site and income could be dropped at any time. This HAS happened to me 2 or 3 times in the past years so folks, bite the bullet and just do it.
Shaun
Excellent. Worked for me first time. No doubt you have saved my life, at some point in the future
Wow, it finally worked…thank you so much for your help with this. I was wondering….can this script be modified to only backup the public_html folder and NOT the entire home directory? I ask this because Hostgator caps Cpanl backups at 50,000 inodes. I had to whittle my site down a LOT to get it below 50,000. Once a site hosted by Hostgator reaches larger than 50,000 inodes, they wont allow full site backups anymore. So if this script was able to backup only the public_html folder, that would help the size limitation. Can this be done?
@Jason,
This particular method is used for full cPanel backups. You can create a script that only backs up a certain directory and create a cron job to run that on a schedule. That’s pretty simple. The issue is that it will not get the MySQL backups, which the full backup does, so you would have to do something different to get the MySQL backups for each of your databases.
Wil
I would be happy to send them to you…but which 2 files…i know you need the backup.sh but what is the other one? Thank you so much for helping me get this going.
@Jason,
Sorry it is not working for you. A LOT of people have used this procedure and got it working without hitches. I am not sure why it is not working for you. When you create the cron job, are you leaving the double quotes in place? If you want to download and zip the two files (change you password in the files, of course), I will be glad to look at them for you. Just upload the zip file to a web accessible directory and post a link here…(I won’t approve the comment to nobody will see your URL).
Wil
That didnt work either, when i did all the copying in IE, it just emailed me a blank email. I think im just gonna give up.
Let me try using IE instead of Firefox.
Thats exactly what i did…i copied it directly from this page into the cpanel text editor.
@Jason,
Make sure the entire script is on one line. Open two browser tabs and copy the script from above and paste it into the editor. Do NOT paste it anywhere else first the copy/paste again.
Wil
Ugh…i am SO close i can taste it…been trying to do automated backups for 2 weeks and just cant get it. I think this site is the answer though.
I am now being emailed this error :
wget: missing URL
Usage: wget [OPTION]… [URL]…
Try `wget –help’ for more options.
@Jason,
Just use the default setting for character encoding in the cPanel text editor, which is probably UTF-8.
Wil
Thanks for the reply, i will try it through the Cpanel text editor…but it is asking what character encoding i want to use and i dont know what to select…sorry, im not a programmer.
@Jason,
Please copy the scripts again from above and recreate on your server. It looks like my new template messed up the double quotes and I had not noticed. This is probably the issue. They are fixed now.
Sorry for the trouble you had with it. Let me know how it goes.
Wil
I finally got emailed this from the script :
/home/********/backup.sh: line 1: {rtf1ansiansicpg1252deff0deflang1033{fonttbl{f0fswissfcharset0: command not found
/home/********/backup.sh: line 1: }}: command not found
/home/********/backup.sh: line 2: {*generator: command not found
/home/********/backup.sh: line 2: }viewkind4uc1pardf0fs20: command not found
/home/********/backup.sh: line 3: syntax error near unexpected token `}’
/home/********/backup.sh: line 3: `}’
@Jason,
Do what I suggested above (recopying the scripts), but this is a different problem all together. In the video I showed you how to use the file editor in cPanel. It looks to me like you pasted it into MS Word or some other word processor and then uploaded because my script is one single line (not 3) and those commands it is complaining about are control characters that your word processing application is adding in. Please delete the scripts on your server and try again using the cPanel web-based file editor as shown in the video.
Wil
I cant get this to work no matter what i try…i followed you instructions perfectly, and the script never executes.
Hey Will
Excellent info.
Q? Will this backup include Add on Domains and/or Subdomains?
Thanks
Paul
Paul,
Glad you enjoyed it. To answer your question, yes, this process will back you entire account (including all the data). Since add-on domains are simply folders inside the public_html directory, it will indeed back those up.
You can see this by downloading the backup file and extracting it on your local machine, so you can see all the data.
Wil
Eric, thanks for posting a link to your script!
So many people think their hosting provider backs up their data… it should be standard, but many have found out the hard way, isn’t.
I have my FTP or Amazon S3 server backup script available for free on my webiste. It’s configured for GoDaddy, but can be changed for any dedicated server.
@JdotA I see what happened. The problem was with WordPress. It converted the -- into — above, so when you pasted it, it cause the problem. I have hard coded -- in the above example, so this will no longer happen to anyone else. Thanks for the update and congrats on getting this going! Now you can sleep better at night knowing your data is safe!
Success!!!! It looks like there were a couple of things causing the backup to fail.
1st: After I pasted and saved the script into the HostGator text editor, some of the “-” were being auto converted into the ascii code for the symbol “–”. The only ones that actually changed were the first “-” in the following: “–http-user=$1″ , “–http-password=$2″ and in “–post-data=”. So I disabled the auto Disable Character Encoding for the HostGator text editor and then it saved correctly.
After that I finally started receiving error msgs when the backup was run. I had not been receiving any msgs before this.
2nd: Once I received the error msgs, I looked up a list of options for wget and realized that I needed to have “–” instead of “-” in front of “–http-user=$1″ , “–http-password=$2″ and in “–post-data=”. Once I made those changes the Backup was successful!
Thanks for all your help and the tutorial!
@JdotA No, Hostgator does not require any special permissions.
Can you double check the file name of the script your created? Is the file called backup.sh?
Is it in the root directory of your account? (at the same level as the public_html directory, not within it, for example)? Can you post a screen shot of your directory somewhere and send me a link on Twitter?
Also, the filename is case sensitive, as is your username and password.
Wil
@Wil Checked both and they are fine, Username & Password each inside their own set of quotes and email without quotes. The script was copied directly from this page. Does host gator require any special permissions in order to run the scripts?
@JdotA Here are a few things to check:
1. Did you leave the USERNAME and PASSWORD in the cron command inside quotes? You should surround those values with quotes (the video does not show this, but the example code does). This is required in case you have special characters such as “&,?,” or others in your password.
2. Did you copy and paste the the backup.sh script exactly as above? The entire script is one single line. If you pasted it elsewhere and used another text editor then uploaded the file, that may be your issue. Make sure you create the file exactly as I showed on the video using cPanel so that there are no special characters (line feeds, carriage returns, etc) in the file. Copy the code, go into the cPanel editor and paste the whole script exactly as it is show in the sample code above.
Wil
After trying a few times, I am still not having much success. The cron job that backup the site is not running no matter what i try. I tested cron by using the mv command above and that worked fine so cron is functioning. Its just not generating the backup. Any other ideas? Thanks.
Thanks for all the info. I am setting it up right now, will post the results later today!
@Adam, thanks for the link (click on Adam’s name above).
Looks like your s3sync solution can easily be used with the tutorial I show here in order to only have 1 file to upload.
In looking at your script, I learned something VERY cool that I was not aware of….the mysqldump switch to dump all existing databases at once. VERY cool!
Wil
I automate my backups to amazon S3 using s3sync. I have a tutorial to it in the url i left ^
@JdotA: Here is what you can do to get around this problem:
1. Create a folder in the root directory called “backupfolder”
2. Create a cron job as shown in the video with the following command: mv ~/*.gz backupfolder
3. Make sure you schedule this for about 30 minutes or so after the backup runs to make sure you backup completes before this runs.
What this will do is use the Linux/Unix “mv” command to move the backup file from the root directory to the “backupfolder” so your FTP client can just download those files.
Don’t forget to also remove the files from the server, otherwise you will eventually run out of disk space and if your FTP client is not smart enough to realize that the old files have already been downloaded, each day your downloads will take longer as more files will get downloaded.
Wil
Thanks again for the great info. I am on OSX and I cannot find an app like AutoFTP to pull down a specific file type. I can use Automator to pull the contents of a folder but if I use that, Automator tries to pull all of the files in the home directory. But since HostGator does not let you change the save location I have to try find some other way of doing it. Any info that you might have would be of great help!
Will,
Excellent job with the explanation. I didn’t realize it was so easy to backup your files. Going to have to try this. Recently I upgraded my phplist through Fantastico, and it caused major havoc. This would have come in handy. I was able to get it restored, but what if I hadn’t.
Thanks.