MPN2DMD

Summary

summary of the MPN Tec

General

We call this application „MPN2DMD“ as a work title

Spec of the process

Content delivery

DMD2 receives new digital audio tracks automatically on a server via FTP. The Data is stored in a defined "INCOMING" directory on the FTP server.

We don't expect more than 200 tracks within 24 hours, but delivery is constantly happening.

Processing new files

All audiofiles are delivered as .flac files. Additionally to every audiofile, a metadata file (.txt) is also delivered.

MPN2DMD:

1. decodes the .flac files to .wav files with flac
2. encodes .mp3 files out of the .wav files according to the config with LAME
3. shrinks existing coverart to max. 1000pix width (height automatically adjusted, keep aspect ratio)
4. tags the .mp3 files according to the config with metadata and also coverart (ID3V2 Spec)
5. normalizes the .mp3 file(s) to the specific value
6. renames and stores the file(s) to the destination directory with the file name masks
7. deletes .wav files and moves original .flac and .txt files to the 'compiled MPN data' folder

Spec of the Config

MPN2DMD does offer certain parameters that can be triggered over a text- or xml based config file.

Following parameters shall be covered by the config file:

Encoding format

• MP3
• OGG
• AAC
• AACp
• MPG (MPEG Layer 2)

(for start only MP3 required)

VBR/CBR

switch VBR yes/no, if no, CBR applies

bitrate

• 128
• 192
• 256
• 320

sampling-rate

• 44100
• 48000

normalizing level

MP3 Gain value (e.g. 90 for 90%)

target directory structure

containing:

Default folder: "\\obelix01\allmusic\mpn_content\" 
Compiled MPN data folder: "\\obelix01\__mpnoriginaldata\"
File name mask: "%G\%G  %A %Y\%T %S"
                e.g. "Madonna\Madonna__True Blue__1988\01__Papa_Dont_Preach.mp3"
                Variables: %G=Artist, %A = Album, %T = Track #, %S = Song, %Y = Year

replace spaces with underline char: Additional flag (see above example)

file rename strings

Since Linux filesystems have some issues with "weird" characters like "ü", "?", etc. all files need to be checked for such characters. We'd like to have a replacement table that advises the system how to treat such characters.

This logic only applies to the filenames, not to the ID3V2 tags!

Replacement table logic: "before|after", e.g.

ä|ae
é|e
!|  (blank)
+|and

Delimiter is "|" - might be another that doesn't occur in any tags

case

• lower
• UPPER
• Capitalize First Letter
• CAPITALIZE first word

applicable to ID3V2 Tags and filenames

special directory name/path

for 'unknown' metafiles/musicfiles that have to be compiled manually (configurable)

Spec of the MPN Data

In order to understand the logic of the MPN data and the metadatafiles, a ZIP file with sample data has been uploaded to this file (54 Megabytes)

The filename is created out of the Artist's name (10 digits), EAN/UPC (13 digits) and ISRC (12 digits):

• ARTIST7890.EAN4567890123.ISRC56789012.flac
  e.g.: GomezHeydi.0743219451026.DEC710200193.flac
  

Metadata are submitted as a semicolon (;) delimited textfile on a per-track basis with the same filename as the FLAC file.

• ARTIST7890.EAN4567890123.ISRC56789012.flac.txt
  e.g.: GomezHeydi.0743219451026.DEC710200193.flac.txt
  

The metadata file contains following fields, separated with semicolons (;):

• ISRC #
• EAN/UPC #
• Disc #
• Track # on Disc
• Title
• Artist
• Playtime
• Filename of FLAC
• Labelcode
• Author(s)
• Publisher
• Titel of Disc
• Label
• Genre
• MPN-Track-Identifier
• Release-Date
• GRID #

Possible ID3V2 Frames

   --AENC    Audio encryption
   --APIC    Attached picture
   --COMM    Comments
   --COMR    Commercial frame
   --ENCR    Encryption method registration
   --EQUA    Equalization
   --ETCO    Event timing codes
   --GEOB    General encapsulated object
   --GRID    Group identification registration
   --IPLS    Involved people list
   --LINK    Linked information
   --MCDI    Music CD identifier
   --MLLT    MPEG location lookup table
   --PRIV    Private frame
   --PCNT    Play counter
   --POPM    Popularimeter
   --POSS    Position synchronisation frame
   --RBUF    Recommended buffer size
   --RVAD    Relative volume adjustment
   --RVRB    Reverb
   --SYLT    Synchronized lyric/text
   --SYTC    Synchronized tempo codes
   --TALB    Album/Movie/Show title
   --TBPM    BPM (beats per minute)
   --TCOM    Composer
   --TCON    Content type
   --TCOP    Copyright message
   --TDAT    Date
   --TDLY    Playlist delay
   --TENC    Encoded by
   --TEXT    Lyricist/Text writer
   --TFLT    File type
   --TIME    Time
   --TIT1    Content group description
   --TIT2    Title/songname/content description
   --TIT3    Subtitle/Description refinement
   --TKEY    Initial key
   --TLAN    Language(s)
   --TLEN    Length
   --TMED    Media type
   --TOAL    Original album/movie/show title
   --TOFN    Original filename
   --TOLY    Original lyricist(s)/text writer(s)
   --TOPE    Original artist(s)/performer(s)
   --TORY    Original release year
   --TOWN    File owner/licensee
   --TPE1    Lead performer(s)/Soloist(s)
   --TPE2    Band/orchestra/accompaniment
   --TPE3    Conductor/performer refinement
   --TPE4    Interpreted, remixed, or otherwise modified by
   --TPOS    Part of a set
   --TPUB    Publisher
   --TRCK    Track number/Position in set
   --TRDA    Recording dates
   --TRSN    Internet radio station name
   --TRSO    Internet radio station owner
   --TSIZ    Size
   --TSRC    ISRC (international standard recording code)
   --TSSE    Software/Hardware and settings used for encoding
   --TXXX    User defined text information
   --TYER    Year
   --UFID    Unique file identifier
   --USER    Terms of use
   --USLT    Unsynchronized lyric/text transcription
   --WCOM    Commercial information
   --WCOP    Copyright/Legal infromation
   --WOAF    Official audio file webpage
   --WOAR    Official artist/performer webpage
   --WOAS    Official audio source webpage
   --WORS    Official internet radio station homepage
   --WPAY    Payment
   --WPUB    Official publisher webpage
   --WXXX    User defined URL link

Mapping

 - isrc = TSRC
 - upc = Txxx (I believe Tag & Rename creates a 'Catalogue Number' field .. see also here)
 - disc = TPOS
 - tracknum = TRCK
 - album = TALB
 - playtime = TLEN (in milliseconds)
 - filename = TOFN
 - labelcode = COMM ('Labelcode:[code]')
 - authors = TCOM
 - publisher = TPUB
 - label = TOWN
 - genre = TCON
 - mpntrackid = UFID
 - releasedate = TYER (only YEAR, not month or date)
 - grid = Txxx  (see also here)

Coverart is submitted in the following format:

• EAN4567890123.front.jpg
  Frontcover in resolution 600x600.
• EAN4567890123.back.jpg
  Backcover in resolution 600x600.
• EAN4567890123.txt
  Product-Fact of corresponding Disc.

Updates

If any of the metadata is updated after the initial delivery, only textfiles (with same filenames than in the first phase) are delivered, but no FLAC-files. Since we don't store FLAC files, those metadata files shall be moved to a special directory for DMD2 staff can manage the changes manually. Path and name of the special directory to be integrated within the MPN2DMD config file.

Pre-requisites

• apt-get install libgd2-xpm and libgd2-xpm-dev
• Perl 5 (installed via apt-get) with the following Perl modules compiled from CPAN (tutorial here: http://www.troubleshooters.com/codecorn/littperl/perlcpan.htm)
  • MP3::Tag
  • GD
  • Proc::Daemon
• FLAC - apt-get install flac
• MP3gain - apt-get install mp3gain
• LAME - installed from source : http://lame.sourceforge.net/index.php

Code Location

All the main code is located here : automatix:/home/mpnupload/work/MPN2DMD

There are only two scripts that are "code", processFLAC.pl, and pollDir.pl.

Configuration Files

System Software Configs

There are no system software configuration that is required.

Application Software Configs

MPN2DMD is controlled by a single configuration file called "process.cfg" and is located here: automatix:/home/mpnupload/work/MPN2DMD/process.cfg

This file specifies all the required paths needed by the script. If the scripts are moved to another location, please make sure this configuration file is updated.

After the configuration file has been updated, you will need to restart the daemon processes. This can be done by simply killing the running processes and restarting them (see the rc.local entries for how to start them).

Daemon Jobs

There are two daemon jobs, and they should be run as the mpn user. The following entries should be added to the rc.local start script.

su mpnupload -c "/home/mpnupload/work/MPN2DMD/pollDir.pl /home/mpnupload/work/MPN2DMD"
su mpnupload -c "/home/mpnupload/work/MPN2DMD/processFLAC.pl /home/mpnupload/work/MPN2DMD"

Note that each process takes a single parameter, which is a directory that contains the corresponding process.cfg file to use.

Cron Jobs

There are no cronjobs for MPN2DMD

Users/Groups Setup

Everything is run under the user: mpnupload group: mpnupload. Nothing should need to run as root.

Security Updates

Security updates are fine, just make sure not to do any Major updates, (like upgrading to Perl 5 to Perl 6)

Client Applications (if any)

None.