Upload Information Tools
From ISP_RAS
All scripts described below use routines from Upload.pm and Distros.pm modules
Format of files processed by upload* script is described on Collect Information Tools page.
Contents |
Prerequisites
In order to run upload_app_data.pl or upload_distr_data.pl you should have perl with DBI module installed.
If database connection parameters are not passed as script arguments, then environment variables $LSBUSER, $LSBDBPASSWD, $LSBDB, $LSBDBHOST will be used.
Download
All scripts described here are available in the unofficial bzr repository. You may also contact ISP RAS team in order to get the latest versions of scripts and any support information.
upload_app_data.pl
Usage: upload_app_data.pl [options] data_file
data_file File with data to be processed
-h|--help Display help message
-d|--DB db_name Database name
-u|--user username Name of user for db access
-p|--password password Password for db access
-m|--host hostname Hostname for DB
-q|--quiet Do not print any information messages
--verbose Be verbose
--atk Force handling of input file as tar.gz obtained from the ATK Manager
Analyzes file with application data. The file can be either the one created by the apptodb.pl or tar.gz created by the ATK Manager. This script collects interfaces required by ELF files (both executable and shared objects) of the application. 'Required' interfaces are those ones that are listed in '.dynsym' section of readelf output, have 'WEAK' or 'GLOBAL' binding, 'DEFAULT' visibility, whose type is 'FUNC', 'OBJECT' or 'COMMON' and whose 'Ndx' field is equal to 'UND' (i.e. undefined).
For each such interface the script checks, if it is provided by application's shared objects. (Interface is treated as provided by shared object, if it is listed in this shared object's readelf output in '.dynsym' section with 'WEAK' or 'GLOBAL' binding, 'DEFAULT' visibility and with 'Ndx' != 'UND'). If this is not the case, then interface is regarded as 'external' one (i.e. this interface should be provided by other libraries of the operating system). All 'external' interfaces are placed into the RawInterface table (if it doesn't contain them yet) and links between all interfaces and the application are inserted into the AppRInt table. If a new record is being inserted into the RawInterface table, upload_app_data.pl tries to guess if an interface belongs to any class and if this is the case, record in RawInterface table is assigned to the appropriate record in the RawClass table (and new record is inserted into the RawClass table, if needed). If binary level symbols indicating class presence are met ('vtable for', 'typeinfo for', etc.) then appropriate record will be added to the RawClass table (but such symbols themselves are not marked as class members).
Application's ELF files may also require some shared libraries. Such libraries are also collected and apprropriate records are inserted into the AppLib table. (Note that only run-time names of these libraries can be obtained from 'Dynamic section' of readelf output).
The script also detects Perl and Python modules used by application. For each found dependecy a record is placed to the AppRILM table.
If database parameters are not set, enviroment variables $LSBDB, $LSBUSER, $LSBDBPASSWD and $LSBDBHOST will be used.
After data upload is completed, please perform the actions described in the Post-upload section below to make your changes visible in Navigator.
Examples
Upload data from APP_data file using environment variables in order to connect to the database
upload_app_data.pl APP_data
Upload data from APP_data file using in the database 'tempdb' on the 'remotehost' host and using environment variables for other database connection parameters
upload_app_data.pl -d tembdb -m remotehost APP_data
upload_distr_data.pl
Usage: upload_distr_data.pl [options] data_file
data_file File with data to be processed
-h|--help Display this help message
-d|--DB db_name Database name
-u|--user username Name of user for db access
-p|--password password Password for db access
-m|--host hostname Hostname for DB
-q|--quiet Do not print any information messages
--verbose Be verbose
This script can be used in order to upload data about standalone component or distribution (actually distribution is a set of components). The behaviour depends on kind of file given to the script - if this file is 'distrtodb.pl' output, then information about distribution will be uploaded, and if this is 'componenttodb.pl' output, information about standalone component will be uploaded.
For components, the script analyzes libraries, commands and interfaces provided by them. The criterion of detecting interfaces that are 'provided' by component is the same as described above for application's shared objects - interface is treated as provided by shared object from distribution, if it is listed in this shared object's readelf output in '.dynsym' section with 'WEAK' or 'GLOBAL' binding, 'DEFAULT' visibility and with 'Ndx' != 'UND'.
For distributions, the script collects upstream components included in the distribution, and for each component procedure described above is performed.
The script fills the tables Distribution, Component, RawCommand, RawLibrary, RawClass, RawInterface, RLibRInt, RLibRInt, RawILModule and CompRILM. If the appropriate record already presents in table, the script will use it insted of creating new one (therefore it will not produce mysql 'Dup entry for key' errors).
If the component is provided by several packages, then package names will be saved in one string, semicolon-separated. If the component already presents, but the one of the component packages is not mentioned, this package will be appended to the list of packages.
If database parameters are not set, enviroment variables $LSBDB, $LSBUSER, $LSBDBPASSWD and $LSBDBHOST will be used.
After data upload is completed, please perform the actions described in the Post-upload section below to make your changes visible in Navigator.
Examples
Upload data from DIST_data file using environment variables in order to connect to the database
upload_distr_data.pl DIST_data
Upload data from DIST_data file using in the database 'tempdb' on the 'remotehost' host and using environment variables for other database connection parameters
upload_distr_data.pl -d tembdb -m remotehost DIST_data
Post-upload Actions
LSB Navigator usually do not use the Community tables (filled in by upload* scripts) directly, since they are too large and it takes too much time to get data from them. Cache tables described at the Database Schema pages are used instead. These tables are created automatically and should be regenerated after every change in the community tables.
To regenerate the table, cd to the specdb directory (where all database files are located) and perform the following actions:
make dumpall make cache
Cache regeneration takes from several minutes to half an hour, depending on machine characteristics and load.