SETTING UP A NEW EXTERNAL GRABBER ------------------------ The internet epg software can only use xmltv-data for which a "content" record has been defined in dbChannelMap.dat. You can do this as follows (see rest of this readme for more details) -If you run the software on relook, first turn on swap: swapon /dev/ide/host0/bus0/target0/lun0/part1 -configure the grabber of your choice. For instance: /opt/bin/tv_grab_se_swedb --configure -create a channel list /opt/bin/tv_grab_se_swedb --list-channels --output /tmp/channels.xml -import the channel list into dbChannelMap.dat /var/media/epg/software/channelMap.php --import-xmltv-channels=/tmp/channels.xml --db=/var/media/epg/software/data/dbChannelMap.dat --grabber=tv_grab_se_swedb -If you haven't done so already, import satellite channels from your db.dat to dbChannelMap.dat. Currently, dbChannelMap.dat contains data for Hotbird, Astra, Astra 2 and Eurobird. If you receive other satellites import their channels as follows /var/media/epg/software/channelMap.php --import-relook=/var/data/db/dat --db=/var/media/epg/software/data/dbChannelMap.dat -You may need some of the other modes (--gues-by-name, --map-channels etc..) to associate the satellite channels correctly with the xmltv content. -finally run: epg.php --setup OVERVIEW ---------- The channelMap database is meant to define globally unique mapping of sattelite channels to specific "content". it also provides a similar mapping of EPG-data sources to specific content. With satellite channel we refer to a specific video-stream (with one or more related audio streams) broadcast on a specific frequency on a specific transponder of a satellite (or on a specific terrestrial frequency). With "content", we refer to what ordinary people would typically call a "tv-channel", i.e., a set of programs broadcast under a certain name. However, we avoid the word "tv-channel" in this document. For example, the content "julie" is broadcast on three different channels on hotbird at the time this document was written. In the channelMap database, content is identified by a 32-character string that looks like a random string. The table SatChannelInfo links this identifier to one or (usually) more channels. This allows an epg program to discover that the 3 julie channels broadcast the same content and can share the same EPG-information. The table ProgramSourceInfo contains similar data, but for epg-data sources. For example, the content called "ZDF" by tv-viewers (more accurately, the content identified using a very specific 32-character string, let us say 01234567890ABCDEF01234567890ABCDEF) epg-data can be obtained in several different ways. For instance, through the xml-tv grabber probrams tv_grab_de_tvtoday under the name 'zdf.de' and using the grabber tv_grab_fr under the name 'C44.telepoche.com'. Furthermore, grabbers can be built that grab EPG-information for this content from teletext. Using ProgramSourceInfo an EPG-system can look up where it can get information about the content 01234567890ABCDEF01234567890ABCDEF. Of course, defining a database is one thing, but getting all the information for it is another. We assume that in practice many people will work together to insert correct information into the database and keep it uptodate. The database structure and the program to update the database was designed to facilitate this cooperation. It is essential that the following conventions are respected, both by people that insert information in the database using SQL-statement, or by programs that update the database -the same content should always be indentified by the same 32-character Content_ID. Never use a new Content_ID except when you discover new content -Never use different Content_ID's for different content. It may be tempting to associate 'BBC1 England' and 'BBC1 Scotland' with the same Content_ID, because they broadcast the same programs most of the time and you prefer to use the same EPG-data for both contents. However, other people might not agree with this, so you cannot pass your "corrections" on to other people, who may actually consider them "errors". If you wish to use the same EPG-info for 'BBC1 England' and 'BBC1 Scotland' then you should handle this at the level of the EPG-program (through a config file), not at the level of the database. -for new content, a new Content_ID must be generated. It is very important that you choose a RANDOM Content_ID for new content. Otherwise you run the risk that someone else in another country used the same Content_ID for different content. Again, this would prevent the exchange of channelMap data and render the database useless. The preferred way to generate ramdon ID's is using the md5sum() algorithm, applied to a random input. -In practice, there will be more channels for which no epg-data can be found than channels for which grabbers exist. For this reason, Content_ID's should only be defined for content for which at least one epg-source exists. In the database, this means that in SatChannelInfo only Content_ID's are allowed which also occur in ProgramSourceInfo. All other Channels should have Content_ID=NULL. Database update programs can overwrite NULL values when they discover new epg-sources, even if they are not 100% sure that the source matches the channel. On the other hand, they should be very careful to overwrite any other value. -Since multiple epg-sources may exist for the same content, but not all of them are equally good, the programSourceInfo table has a Priority column for each content/epg-source combination. EPG-programs should normally use the source with the highest priority. Nevertheless, it is very useful to have more than one source per content, just in case the best epg-source fails (it happens a lot!). -SatChannels and ProgramSources also have a human-readable "Name". On the one hand, this is handy when using SQL-commands to view or update the database; on the other hand the names can also be used by programs to "guess" the correct Content_IDs for channels. It is safe to change a name into a more meaningful one (e.g., remove rubbish characters). The channelMap program is a program that updates the database. It recognizes the following commands *importing channels from an xmltv-grabber's output --import-xmltv= --grabber= --priority= xml-file is a file produces by, e.g. tv_grab_fr. It is essential to specify the grabbername (in the example: tv_grab_fr), because this will allow you to share your database with other people. Setting a priority is only recommended when you have some reason to believe that the grabber is better or worse than others. Note that channelMap.php makes some assumptions, e.g., that programsources with the same xmltv "channel-id" or with the same display-name are for the same content. Usually these assumptions are correct, but you can use other commands to correct any errors. *Importing channels from the relook db.dat database --import-relook= [--update-names] This searches db.dat for new channels, identified by Satellite position, transponder frequency and polarisation, Network_ID and Program_ID (the information is somewhat redundant; in principle Network_ID and Program_ID should be sufficient, but in practice they are not). New channels are inserted into the channelMap database. Old channels are NEVER overwritten. NOT IMPLEMENTED: With the option "update-names" the "Name" of the channels are overwritten with the ones in db.dat but all other information is kept. This may be useful when channels initially are named "test 123" and later get a real name. However, "update-names" is probably to dangerous as it overwrites ALL names of all channels *guessing associations between channels and content. --guess-by-name This command examines all channels with Content_ID=NULL. It then looks for programsources with similar names and if it finds them, it uses their Content_ID. For instance, if a satellite channel named "zdf" exists and tv_grab_de_tv provides an xmltv-channel named "ZDF" then it associates "zdf" with the same content as "ZDF". While this works in many cases, it will also introduce quite a few errors (e.g., may channels are called "tv1", because the broadcasters lack imagination when choosing a content name). It is recommended to use --guess after adding programsources or satellite channels. You can then correct the errors using other commands. --guess-by-name is safe because it will never overwrite existing information. The "guessed" column in SatChannelInfo indicates whether the Content_ID field was updated by guessing or by a more clever method. Important note: currently "guessing" is implemented as follows: names of channels and programsources are normalised by converting them to lower case and eliminating spaces and certain characters. Two names match when they have the same normalised name. *correcting channel mappings --map-channels= [--dbdat=db.dat] [--guess] [--overwrite] file contains a number of lines in one of the following formats: 'zdfd' => 'xmltv:tv_grab_de_tvtoday:zdf.de' Map all channels with name 'zdfd' in SatChannelInfo to the Content_ID of xmltv:tv_grab_de_tvtoday:zdf.de 507 => 'xmltv:tv_grab_de_tvtoday:zdf.de" Map the channel with number 507 on your relook (column ch_order in ChannelInfo of db.dat) to the Content_ID of xmltv:tv_grab_de_tvtoday:zdf.de Of course, you must provide --dbdat=... for this to work. Note the absence of quotes around the number 507. If you use quotes, the mapping is for the channels named 'to7' rather than for the channel with number 507. 507 => 'ecccd44dbc666232e60467698642db43' Same as above, but the right-hand side is interpreted as a Content_ID. 'zdfd' => NULL 507 => NULL This can be used to delete existing mappings so that they can be guessed afterwards. 'xmltv:tv_grab_uk_rt:uk.scifi.com' != 'xmltv:tv_grab_de_tvtoday:premscifi.tvtoday.de' This can be used when distinct contents have been mapped to the to the same Content_ID by mistake. uk.scifi.com is assigned a new Content_ID and all existing SatChannels are now linked to premscifi.tvtoday.de Note: if filename="-", then data is read from standard input instead of from a file The --guess option, when present, not only updates the channels with the exact name "zdfd" but also channels with similar names, such as "zdf-d" or "zdf d". The --overwrite option, when present, will update ALL matching channels. Without it, only channels with Channel_ID=NULL are updated *NOT IMPLEMENTED correcting programsource information --map-sources=file file contains lines in the following formats: xmltv:tv_grab_fr:telepoche.com=xmltv:tv_grad_detvtoday:zdf.de This indicates that the two program sources refer to the same content. The Content_ID of the left source is replaced with the Content_ID of the right source in all tables. The information is also recorded in a table "ContentAliases" xmltv:tv_grab_fr:telepoche.com!=xmltv:tv_grad_detvtoday:zdf.de This indicates that the two program sources refer to DIFFERENT content. A new Content_ID is generated for the left source. The information is also recorded in a table "ContentAliases". SatChannelInfo is NOT modified xmltv:tv_grab_fr:telepoche.com=NULL This indicates that the record xmltv:tv_grab_fr:telepoche.com should be removed from the database. If there is no other record in ProgramSourceInfo with the same Content_ID, then not only the program source is removed, but also the corresponding content. This is done by setting Content_ID=NULL for all sattelite channels that have the same Content_ID as the deleted xmltv:tv_grab_fr:telepoche.com program source *NOT IMPLEMENTED importing records from another channelMap database. --import-channel-map=foreigndb.dat New programsources and new satellite channels are copied from foreigndb.dat to channelMap.dat. Also, if satellite channels have Content_ID=NULL in channelMap.dat and a non-zero Content_ID in foreigndb.dat, the Content_ID is updated. Note that importing will never correct erroneous mappings in channelMap.dat. It only inserts new data Database updates ----------------- If you update dbChannelMap.dat, then please share your updates with the community. At a later stage, the channelMap.php program will allow importing data from one dbChannelMap.dat database to another and perhaps even logging of changes. For the time being, please extract new records from the ProgramSourceInfo and SatChannelInfo tables in dbChannelMap.dat as sql statements (if you know how to do that), or make your database public and let us know where to find it on the relookplaza.nl or satellites.co.uk relook forums.