Documentation for the ISOCREATOR Vimeo Software ############################################################################### # OUTLINE ############################################################################### 1: Introduction to isocreator for Vimeo 1.1: Requirements 1.2: Installation 1.3: Integrating isocreator with Vimeo 1.3.1: Using XML 1.3.2: Using SQL 2: Overview 2.1: options.xml 2.2: text_menu_options.xml 2.3: thumbnail_menu_options.xml 2.4: isocreator.sql 2.5: requests.py 2.6: isocreator.py 2.7: generate_xml.py 3: Development 3.1: extending isocreator 4: Unsorted Notes 4.1: Aspect Ratios and Widescreen Clips 4.2: Fonts and Color for Menus 4.3: Acceptable File Types 4.4: Animated Thumbnail Menus 4.5: Custom thumbnails ############################################################################### # 1: Introduction to isocreator ############################################################################### Isocreator is a tool developed for Vimeo that generates DVD ISO files. The software can get information about what ISOs to create from either XML documents, or an SQL database. Isocreator is meant to be run on a server - it is not a client-side application. The software's purpose is to allow users of Vimeo to create DVD videos from their favorite clips on Vimeo. A small overview of isocreator's program flow would look like this: read in options get list of requests for each request: download each requested clip encode each requested clip create an ISO file of the clips email the user a link to the ISO file check for more requests Isocreator does not come with a frontend. The Vimeo staff will have to provide a GUI which sends requests to the isocreator script to process. We have tried to make it as easy as possible to submit a request to the isocreator software. Documentation follows. ############################################################################### #1.1: Requirements ############################################################################### Linux-based operating system Python 2.5 (or greater, should work fine with 2.6) MySQLdb (may be included in your distro's python - on Ubuntu, apt-get install python-mysqldb) On some systems, you may need to install the gsfonts package tovid - http://tovid.sourceforge.net You can also not use MySQLdb if you never plan on using the SQL interface - just comment out the import line in isocreator.py ############################################################################### #1.2: Installation ############################################################################### Unzip the files into a folder on your server. >>Install and patch tovid (http://tovid.sourceforge.net), and then copy the new >>version of todisc provided with isocreator to wherever you installed tovid to - >>/usr/local/bin by default Customize the options files. The default options should be fine. You will have to set the following in options.xml... scheme: xml or sql, respectively smtp_server: if you are using a separate server for sending email, or if the server you're installing it on can't send email. smtp_sender: set this to the email address you want emails to appear from smtp_subject: set these to what you want the subject to be in emails smtp_message: set these to what you want the message to be in emails db_server db_user db_pass: if you're using SQL db requests_table clips_table: if you're using SQL, and not using the provided isocreator.sql file for creating the database url_base: the url to the directory on your server this is extracted to, e.g http://yoursite.com/isocreator/ You will also want to set some options for DVDs with menus. See sections 2.2 and 2.3 for more information on customization of the DVD menus. ############################################################################### #1.3: Using isocreator ############################################################################### We recommend starting isocreator with the command "nohup python isocreator.py &" This allows you to start through ssh, logout, and have the script continue to run. There is also a lot of output generated by the script, and you probably don't want this scrolling all over the screen. You will probably want to set up a cron job to remove files from the working directory that are more than a few days old, unless you want to fill up your server with completed iso files. An example: 0 0 * * * find /path/to/isocreator/working/ -mtime +2 -exec rm {} ; This would check every day to see if there were any files under the /path/to/isocreator/working/ directory older than 2 days, and then delete them. Note that we might keep these files around, or move them to your Amazon S2 account, for the digital locker features in a future version of this DVD ISO software. If you want background images for your DVDs, put the image in isocreator's directory and set the appropriate options in text_menu_options.xml and thumbnail_menu_options.xml See sections 2.2 and 2.3 for more information on customization of the DVD menus. ############################################################################### #1.3.1: Using XML ############################################################################### There is a directory named requests. XML files should be written to this directory. One XML file per request. The structure should look like this: user@example.com text menu dvd thumbnail ntsc test clip one http://example.com/video.avi http://example.com/video.jpg bllah blah test clip two http://example.com/video2.avi http://exaple.com/video2.jpg blah blah blah We believe this is a very simple configuration. Write the XML to the requests directory. When isocreator is done processing a request, it will move its XML file into the finished directory. ############################################################################### #1.3.2: Using SQL ############################################################################### If you do not want to send XML requests to the script, you can also request DVD ISO files to be made using SQL. Assuming you are using the provided isocreator.sql scheme, you would do something like this to add requests (in php): mysql_connect($server,$username,$password) mysql_select_db($db) $query = "INSERT INTO requests VALUES('','$email','$dvd_title', '$menu_type', '$encoding');"; mysql_query($query); $id = mysql_insert_id(); mysql_query("COMMIT;"); //for each clip: $query = "INSERT INTO clips VALUES('','$id','$clip_title', '$clip_url', '$thumbnail', '$description');" ; mysql_query($query); mysql_query("COMMIT;"); There is a more detailed description of the database in section 2.4 ############################################################################### #2: Overview ############################################################################### This section is a technical overview of all the important files included in isocreator. This section should clear up some implementation questions you may have. ############################################################################### #2.1: options.xml ############################################################################### This file contains over-arcing options for the script. Most of these options can be left standard. Available options include, scheme: this can be either sql or xml, and needs to be set to whichever way you are going to be sending requests to isocreator db_server: if you're using sql, the database server. "none" otherwise db_user: if you're using sql, the user to connect to the database as db_pass: if you're using sql, the user's password to use db: if you're using sql, and aren't using the provided isocreator.sql file, set the name of the database here. Leave it alone otherwise requests_table: if you're using sql, and aren't using the provided isocreator.sql file, set the name of the table to use for requests here. Leave it alone otherwise clips_table: if you're using sql, and aren't using the provided isocreator.sql file, set the name of the table to use for clips here. Leave it alone otherwise requests_dir: if you're using xml, and don't want to use the default place for storing unprocessed requests, set it here. Leave it alone otherwise. finished_dir: if you're using xml, and don't want to use the default place for storing finished requests, set it here. Leave it alone otherwise. (You could set this to /dev/null/ if you don't want to keep finished requests) errors_log: if you want to use a different name for logging errors while making requests, set it here. encoded_prefix: if you want to change the prefix used for encoded file names, set it here smtp_server: your smtp_server. change it if it's not localhost smtp_sender: the address you want mail to appear from smtp_subject: subject for emails smtp_message: what you want to say in the emails. working_dir: if you want to change the directory that isocreator does it's work in, change it here url_base: the url to the directory on your server this is extracted to, e.g http://vimeo.com/isocreator/ >>use_thumb: whether or not to enable the use of custom thumbnails for clips in dvd menus. >>thumb_length: when use_thumb is true, the thumbnail specified for a clip is downloaded and turned into an mpeg stream to be used in the menu. This is, unfortunately, necessary (see notes section at the end for a more thorough explanation). Note that this clip is only used for making thumbnails - it will not appear during dvd play. This option is how long, in seconds, to make the clip from the thumbnail image. It's provided mainly for speed tweaking - be aware that values under 3 may cause this to be a bit unstable though. >>default_thumb: if you want to provide a default thumbnail to use if one isn't provided in a request, specify it here. Image should ideally be stored in the same directory as isocreator. ############################################################################### #2.2: text_menu_options.xml ############################################################################### These are options isocreator.py will use when creating DVDs with text menus background_image: image to use as the background for DVDs. This should scale well to 720x480 or 720x576 and should be placed in the same directory as isocreator.py All standard image types, including JPG, GIF, and PNG, are acceptable. crop_or_scale: whether to crop or scale the background image. We recommend scale font: font to use for menu text. Can use any font available on your server. You may need to install additional fonts in your Linux environment. fontsize: fontsize to use for menu text align: how to align the text on the screen text_color: the color to use for text select_color: the color to use for selected text (when someone hits play on a clip) highlight_color: the color to use for highlighted text ############################################################################### #2.3: thumbnail_menu_options.xml ############################################################################### These are the options isocreator.py will use when creating DVDs with thumbnail menus background_image: image to use as the background for dvds. This should scale well to 720x480 or 720x576 and should be placed in the same directory as isocreator.py All standard image types, including JPG, GIF, and PNG, are acceptable. menu_font: font to use for menu titles. Can use any font available on your server. You may need to install additional fonts in your Linux environment. menu_fontsize: fontsize for menu titles thumb_font: font to use for thumbnail titles thumb_fontsize: fontsize for menu titles title_color: color to use for the menu title use_fading: whether or not to fade the menu in and out (note, this will make it take a lot longer to create the menu) thumbnail_shape: the shape to use for thumbnails. Options are normal (square), oval, plectrum (guitar pick), and egg. use_3d: whether to use pseudo-3d thumbnails playall: whether to add a "playall" button to the menu ############################################################################### #2.4: isocreator.sql ############################################################################### This is the dump for the reccomended database structure if you're using the sql scheme. It creates a database called "isocreator" with two tables on it: requests: id email dvd_title menu_type encoding processed clips: id request_id title url thumbnail description clips->request_id is a foriegn key that corresponds with requests->id The tables use the innoDB storage engine - your host may not support this, and you should check with them before deciding to use the sql theme ############################################################################### #2.5: requests.py ############################################################################### This file, included by isocreator.py defines a few classes used by the script. Clip - a representation of an individual clip in a request Request - A base class, representation of one request XmlRequest - Inherited from Request, knows how to get its info from xml files SqlRequest - Inherited from Request, knows how to get its info from an sql database ############################################################################### #2.6: isocreator.py ############################################################################### The main script. This reads in options files, creates appropriate objects depending on what scheme is being used, downloads and encodes clips, creates ISO files, and emails the user. ############################################################################### #2.7: generate_xml.py ############################################################################### This can be used to make a bunch of xml files if you want to test the script, or just see example xml files. Open the file and near the top there's a line that says movie_urls = [] fill this with the clips you'd like to use to test with. e.g.: movie_urls = ['http://www.example.com/vid1.avi', 'http://www.example.com/vid2.wmv'] at the bottom of the script is says: for i in range(0,10) change 10 to whatever number of xml files you want to create Note that the XML files created by this are very rough and for development testing only. ############################################################################### #3: Development ############################################################################### ############################################################################### #3.1: Extending isocreator ############################################################################### New schemes can be made by inheriting from the Request class in requests.py and implementing the functions get_request() and get_clips(). For instance, let's say we want to read from a CSV file. class CsvRequest(Request): def __init__(self,get_from): Request.__init__(self,get_from): def get_request(self): #read in the user's information from the csv file #which should be self.get_from def get_clips(self): #populate the self.clips list with the clip informatio After creating the new class, you need to add appropriate code to the get_requests() method in isocreator.py and also pretty much anywhere in the code you see "if options['scheme'] - this is only three or four places, implementing new ways of retrieving requests should be relatively painless. ############################################################################### #4: Notes ############################################################################### ############################################################################### #4.1: Aspect Ratios and Widescreen Clips ############################################################################### In options.xml - aspect - this can be set to an aspect ratio (e.g. 4:3) or "auto" if set to an aspect ratio, the script will encode clips to said aspect ratio. if set to auto, it will keep the clips original ratio and letterbox them in the dvd. We have successfully made a few dvds with the "auto" feature, and had widescreen and odd-aspected clips letterboxed correctly. The dvd is forced 4:3 right now (but the clips aren't), so things get letterboxed if they are widescreen. We can add an aspect ratio chooser option for DVDs as well, but I don't know of many tvs that aren't 4:3. And I don't think it's an option that would be used very often. (and it would have to be on a per-request basis, which makes it a bit odd for automation) For now I think this meets Jakob's requirement that "Every video should be as large as possible, but cannot be cropped or stretched." ############################################################################### #4.2: Fonts and Color for Menus ############################################################################### Don't be surprised if your results for "convert -list type" are slim. Most web servers will not have any fonts installed (no graphical environment). Let us know if you need help installing fonts on your server. The list of available colors in "convert -list color" should be huge. ############################################################################### #4.3: Acceptable File Types ############################################################################### We have tested making DVDs with wmv, avi, mov, mpg, and rm files. ############################################################################### #4.4: Animated Thumbnail Menus ############################################################################### When making a thumbnail dvd, the script seeks to two seconds into the movie, and makes a small movie clip to use as a thumbnail. In thumbnail_menu_options.xml, thumb_type can be set to static or animated. using static thumbnail menus increases the speed of creating DVDs dramatically. Running a simple test with static thumbnails, the ISO creation process took ~ 1:13. With movie clip thumbnails in the DVD menu, the ISO creation process took ~ 23:45. Please note that we cannot substitute the Vimeo thumbnail GIF or JPG for the DVD menu thumbnail. The thumbnail has to be generated during the DVD creation process. ############################################################################### #4.5: Using your own thumbnails ############################################################################### There is an option now in options.xml - use_thumb, which can be set to true or false. When enabled, this option will do a few things: For thumbnail menus: Use the thumbnail specified for a clip as the thumbnail in the menu. This thumbnail will be shown before the clip when watching the movie. For other types: Insert the thumbnail for a clip before the clip NOTES FOR INTEGRATION- Clip length needs to be present for all of your source files. >> #################################################### # On using custom thumbnails ##################################################### Alright, as of now, thumbnails can be specified without adding an extra clip to the dvd itself. An extra clip still does get generated though - just an mpeg stream of the still frame, which gets swapped for the original clip in the part of todisc that searches through a clip to create thumbnails for it. It should be possible to modify todisc to accept just an alternate thumbnail image and not a mpeg stream created from an image - this would be a pretty big speed up, minute per few clips or so. If you know BASH scripting, open up todisc, and search for #XXXXXX to see the relevant modifications made to todisc that allowed swapping of videos searched through for clips.