User:Daveh/ESO Update
The UESPWiki – Your source for The Elder Scrolls since 1995
This page details the procedure of updating the UESP data after a major ESO update/release.
Contents
- 1 General Prep and Notes
- 2 Export Game Data
- 3 Update Log Parser
- 4 Update esofiles.uesp.net
- 5 Update esolog/esoCommon.php
- 6 Skills and Champion Points
- 7 Update Skill Coefficients
- 8 Mine Item Data
- 9 Update Log Viewer (PTS Only)
- 10 Update Maps
- 11 Update Game Data
- 12 Update Icons
- 13 Update API
- 14 Update Sales Items
- 15 TODO
- 16 Notes
General Prep and Notes[edit]
-
- Run ESO patcher to get latest version.
- (Optional) Edit content3:/home/uesp/esolog/esolog.index and increment/note the log index. Optional but useful if you need to find a specific entry in the new logs.
- (Optional) Backup/delete the uespLog_backupData.txt file created by uespLogMonitor.
- PTS/Live updates are slightly different. Most steps are for both updates while some steps are for only one or the other.
- Note that the VERSION field used for scripts/databases is just the update number itself or the update number with pts appended to it. For example:
-
- Live: 17
- PTS: 17pts
Export Game Data[edit]
-
- Edit createNewExport.sh and update the top 4 lines with the relevant update numbers
- Example for Live Update (using update 17):
VERSION="17" # Game update number ISPTS="" # 1 for PTS, blank for Live LASTVERSION="16" # The last game update LASTPTSVERSION="17pts" # The last PTS game update
-
- Example for PTS Updates (using update 17pts);
VERSION="17pts" # Game update number ISPTS="1" # 1 for PTS, blank for Live LASTVERSION="16" # The last game update LASTPTSVERSION="16pts" # The last PTS game update
-
- Run createNewExport.sh (typically takes 4-6 hours to run, creating 1000k files, using 200GB).
- Fix/address any errors that occur during the script.
Update Log Parser[edit]
-
- Edit the parseLog.php file.
- Update the START_MINEITEM_TIMESTAMP and real-time timestamps.
- Update the MINEITEM_TABLESUFFIX and SKILLS_TABLESUFFIX constants with the current version.
- (Optional) Update the $IGNORE_LOGENTRY_BEFORE_TIMESTAMP1 field with the relevant timestamp. Typically this is done in conjunction with clearing the logEntry table every 6 months or so to prevent that table from becoming too large and slowing down parsing.
Update esofiles.uesp.net[edit]
-
- Create new directory files1:/shared/uesp/esofiles/update-XX/
- Upload all ZIP files created in the export process.
Update esolog/esoCommon.php[edit]
-
- Add API version to $APIVERSION_TO_GAMEUPDATE and $APIVERSION_TO_GAMEVERSION arrays.
- Update the GetEsoItemTableSuffix() function with the latest update numbers.
- Live: Update the GetEsoItemTableSuffix() function to return the correct current update number (usually done after item mining is completed).
- Live: Update the GetEsoUpdateVersion() function to return the current update number (usually done after item mining is completed).
- Update set indexes. Copy content from the Sets.php file generated by the data extraction script and copy into esoCommon.php replacing the existing $ESO_SET_INDEXES variable.
- Optional: Edit and run updateSetIds.php if you need to update the internal game set ID in the setSummary table after you run createSetSummary.php or in another table.
Skills and Champion Points[edit]
-
- Only dump data on a character with no CPs/skills purchased and wearing no items that could affect the mined data. Also any add-ons that modify skill icons (ex: LuiExtended) should be disabled.
- /uespdump skills abilities VERSION
- /uespdump skills types VERSION
- /uespdump cp VERSION
- Upload saved variable files.
- Run log parser.
- Only for Update 28 and Earlier: Edit createCpFits.php with the new table $TABLE_SUFFIX and run.
- Edit fixSkillsSlotted.php with the new table $TABLE_SUFFIX and run.
- Edit createSkillTree.php with the new table $TABLE_SUFFIX and run.
- Edit fixDestructionSkills.php with the new table $TABLE_SUFFIX and run.
- Edit fixPvpArtifactSkills.php with the new table $TABLE_SUFFIX and run.
- Update and run php71 generateSkillIcons.php to create skill icons for tooltips API (after icons are uploaded).
- For Live updates change the current symlink to the new update directory:
ln -sfn ./41 /mnt/uesp/esogameicons/uespskills/current
-
- Update raw skill data and tooltips:
-
- The first step can be done as soon as the files are extracted from eso.mnf.
- Run TestSkillFormat to create the esoRawSkillData.php file from the skill data in eso000.mnf (modify the one input parameter to be VERSION).
- Copy to the esolog/esoRawSkillDataVERSION.php file.
- Update table suffix at top of parseRawSkillData.php.
- The next step has to be done after the skill coefficients have been updated and fixed (see below section for details).
- Run parseRawSkillData.php. Pay attention and fix any warnings/errors displayed (I usually only fix player skills afterwards from the database since there tends to be a lot of minor errors).
-
-
- Run the following commands in ESO to update the flag data:
-
- Reset all CP/skills and remove equipment.
- Buy CPs needed for the various commands below. Remember to unequip prior CPs at each step.
- Run: /usc savedesc (this may need to be done at multiple points)
- Equip the +DOT Damage CP and run: /usc checkdesc dot
- Equip the +DIRECT Damage CP and run: /usc checkdesc direct
- Equip the +ST Damage CP and run: /usc checkdesc stdmg
- Equip the +AOE Damage CP and run: /usc checkdesc aoedmg
- Equip the +DS Duration CP (Shield Master), equip and run: /usc checkdesc ds
- Equip the +DS CP (Bastion) and run: /usc checkdesc ds
- Buy the +HEALING DONE CP and run: /usc checkdesc heal (keep this purchased during the next healing steps)
- Run: /usc savedesc
- Equip the +HOT Healing CP and run: /usc checkdesc hot
- Equip the +AOE Healing CP and run: /usc checkdesc aoeheal
- Equip the +ST Healing CP and run: /usc checkdesc stheal
- Not Needed Since Update 32: Buy the DK World in Ruin passive and run: /usc checkdesc flameaoe
- Equip the Elf Bane set and run while in combat: /usc checkdesc elfbane
- Equip the Diamond’s Victory set and run while in combat (trigger with a ranged spell): /usc checkdesc melee
- Upload log and parse.
-
-
-
- After Items Mined: Run: findSetBonusSkills.php to link scaling sets to their matching skill.
-
-
- (Live Only) Move old tables (minedSkills, skillTree, skillTooltips, minedSkillLines, cpSkills, cpSkillDescriptions, cpDisciplines) to tableOLDVERSION and new tables to tables with no version suffix, for example:
RENAME TABLE minedSkills TO minedSkills16; RENAME TABLE minedSkills17 TO minedSkills;
-
- Can now do this using the script (VERSION is the current update version):
php renameTables.php skill VERSION
Update Skill Coefficients[edit]
-
- Only dump data on a character with no CPs/skills purchased and wearing no items.
- When equipping items only use items/sets with "simple" modifiers. For example, a set that increases spell damage of Fire based skills will affect skills differently and doesn't update the character's weapon/spell damage stat and may result in inaccurate coefficients. Prefer items/sets that just modify base stats and apply to everything. Avoid any +healing, +damage shield, and any cost modifiers (including traits).
- Add all skills that have a description containing a number:
/usc addall
-
- Use an add-on like Alpha-Gear to save item setups. On PTS this doesn't always work since characters are copied/deleted frequently.
- Equip a complete setup of Light Armor and run the command:
/usc save
-
- Wait for the save command to finish before running the next command.
- Unequip 2 random pieces of armor, save, and repeat until naked. This gives around 6 points of data for each Light/Medium/Heavy setup which is a good balance between too few points (less accuracy) and too many points (more accuracy but time and space consuming).
- Repeat with a complete setup of Medium and Heavy armor.
- Run the command:
/usc calc
-
- Reload the UI. Copy/backup the (large) saved variable file.
- Run the command:
/usc reset
-
- Reload the UI and upload the (smaller) saved variable file to the parser. Parse the new data.
- Check for coefficients that didn't compute correctly. This regularly occurs due to skills being updated/changed. Easiest way to directly query the database using a query like:
SELECT id, name, rank FROM minedSkillsXX WHERE isPlayer=1 AND R1<0.99 AND R1>0;
-
- Repeat the query for R2...R6. Note that R values from 0.90-0.99 might be considered "valid" or "good" in some cases.
- Fix the skill coefficient data found in the uespLog.SKILLCOEF_SPECIALTYPES variable within uespSkillCoef.lua as needed.
- Restore the large saved variable backup file in order to recalculate coefficients (or simply redo all steps from scratch).
- Recalculate the skill coefficients by running the commands:
/uespreset log /uespreset temp /usc calc
-
- Repeat the backup/upload/parse procedure until all bad coefficients have been fixed.
Mine Item Data[edit]
-
- Make sure you do this on a max level character (CP160) with no CP/skills purchased and not wearing any ability altering equipment.
PTS[edit]
-
- Only the min/max item data are exported in PTS updates to minimize the time and storage required.
- Initialize the item summary data (used for detecting item data corruption):
/umi createsummary /umi checksummary
-
- If there are any issues detected in the last command you may need to restart the client and re-create the summary data.
- Run the following in-game commands:
/umi table pts /umi autostart 1
-
- After each automatic /reloadui copy the uespLog.lua saved variables to a new file for later manual upload (or do it automatically with uespLogMonitor).
- Repeat the mining for potion data.
/ud uespLog.MinePotionData() /umi table safe /umi potion on /umi autostart 0
-
- Item mining for PTS takes about 2 hours and there should be around 16 saved variable files.
- Upload files and parse.
- Edit createMinedItemSummary.php, update the $TABLE_SUFFIX variable with the current version and run.
- Edit createSetSummary.php, update the $TABLE_SUFFIX variable with the current version and run.
- Edit updateSetIds.php, update the $TABLE_SUFFIX variable with the current version and run (do this after updating $ESO_SET_INDEXES).
Live[edit]
-
- Initialize the item summary data (used for detecting item data corruption):
/umi createsummary /umi checksummary
-
- If there are any issues detected in the last command you may need to restart the client and re-create the summary data.
- Run the following in-game commands (make sure level/type/item type/potion options are turned off):
/umi table safe /umi autostart 1
-
- Note that the entire process takes around 1 week of full time mining to complete.
- Repeat the mining for potion data.
/ud uespLog.MinePotionData() /umi table safe /umi potion on /umi autostart 0
-
- Save the valid item data by the command (replace XX with the current version):
/umi idcheck XX
-
- Edit createMinedItemSummary.php, update the $TABLE_SUFFIX variable with the current version and run.
- Edit createSetSummary.php, update the $TABLE_SUFFIX variable with the current version and run.
- Edit updateSetIds.php, update the $TABLE_SUFFIX variable with the current version and run (do this after updating $ESO_SET_INDEXES).
- Delete fixitems.lua.
- Update version in findMissingItems.php and run.
- Copy output found in fixitems.lua to uespLog.lua and run /ud uespLog.StartMineTest(1, false) in-game to mine those missing items. Upload log, parse and re-run item summary scripts.
- Repeat for findBadItemSets.php, findBadItemSummaries.php, findBadItemSetDescs.php, and findBadItemNames.php until no more bad items are found.
- Run: findSetBonusSkills.php to link scaling sets to their matching skill.
- Move old tables (minedItem, minedItemSummary, setSummary) to tableOLDVERSION and new tables to tables with no version suffix, for example:
RENAME TABLE minedItem TO minedItemXX; RENAME TABLE minedItemYY TO minedItem;
-
- Can now do this using the script (version is the current update version):
php renameTables.php item VERSION
-
- You may need to restart Apache on content3.
Update Log Viewer (PTS Only)[edit]
-
- Only needed for PTS updates.
- Edit viewLog.php and change the ENABLE_PTS_VERSION constant to the current update version.
Update Maps[edit]
-
- Create a 400% for all combined maps (this can be done any number of ways, the PhotoShop version is given below):
-
- (Do Once) Create an action to resize 400% using "Preserve Details 2.0" resampling with 100% reduce noise.
- Run an Image Processor using the previous action on all images in goodimages-VERSION\CombinedMaps and output to goodimages-VERSION\LargeMaps.
- Typically takes 2-3 hours to generate all maps.
- Usually only have to do new/updated maps found in CombinedMapsNew or CombinedMapsNewPts.
- Create the zoom levels 11 and 12:
-
- Run the Python script: CreateTilesFromLargeMap.py VERSION e:\esoexport\
- Copy tiles to NewMaps: ./export/CopyNewMapTiles1112.sh VERSION
- Upload any files in the NewMaps folder to content1/2/3:/home/uesp/www/esomap/ and files1:/shared/uesp/maps/esomap/.
rsync -avz ./ dave@content3.uesp.net:/home/uesp/www/esomap/ chown dave:uespadmin -R /home/uesp/www/esomap/ chmod g+rwx,o-w -R /home/uesp/www/esomap/ rsync -avz /home/uesp/www/esomap/ dave@content1.uesp.net:/home/uesp/www/esomap/ rsync -avz /home/uesp/www/esomap/ dave@content2.uesp.net:/home/uesp/www/esomap/ rsync -avz /home/uesp/www/esomap/ dave@files1.uesp.net:/shared/uesp/maps/esomap/
-
- Edit maps_new.txt that contains a list of all new maps to add.
-
- Edit the second column to be the displayed name.
- Edit the last column to be the ID of the parent world. For new worlds change this to be -2 so the database can be manually updated later. Otherwise enter the worldId as found from the ESO map if applicable.
- On db1 find the last user world ID by the SQL command: select max(id) from uesp_gamemap.world;
- Edit createNewMaps.sh with the first free world ID (+1 from the previous step) and update version and run.
- Upload the newmaps.sql to db1.
- Backup the current uesp_gamemap database:
mysqldump --opt -u user -p uesp_gamemap > gamemap.sql
-
- Run the SQL file:
mysql -u user -p uesp_gamemap < newmaps.sql
-
- Run php makeZoomLinks.php on files1 to update Leaflet symlinks for the new map.
- Test the map to ensure new maps have been successfully added.
Update Game Data[edit]
Achievements[edit]
-
- This is usually only done for Live updates.
- /uespdump achievements VERSION
- Upload log and run log parser.
- Copy old esoAchievementData.php to esoAchievementDataXX.php
- Run createAchievementData.php
Collectibles[edit]
-
- This is usually only done for Live updates but can be done for PTS updates in order to fully update the uespLog runebox data.
- /uespminecollect
- /ud uespLog.MineCollectibleTree()
- Run createRuneBoxData.php (after mined items are updated) and update uespLog.RUNEBOX_COLLECTIBLE_IDS array in uespLog.lua with any new runebox/collectible pairs.
- Upload log and run log parser.
- (Live Only) Copy old esoCollectibleData.php to esoCollectibleDataXX.php. Copy data out of tempData in the uespLog.lua saved variable file and re-format/paste into esoCollectibleData.php.
Books[edit]
-
- This is usually only done for Live updates.
- Perform this on a character that has unlocked Eidetic Memory/Shalidor's Library.
- Run /ud uespLog.MineBookTree() in-game.
- Run /ud uespLog.MineBooks() in-game (upload/parse log data).
- Copy data from the tempData section of the saved variable file, re-format, and paste into esoBookCollectionData.php.
- Upload books.sql to db1 and run.
Quests[edit]
-
- This is usually only done for Live updates.
- Update the esoQuestData.php file from the Quests/Quest.php file created by the export script.
- NOTE: This step is no longer required due to a change in quest parsing.
-
- Upload quests.sql to db1 and run.
Recipes[edit]
-
- This is usually only done for Live updates.
- Perform this on a PTS template to ensure all recipes are recorded correctly.
- Run /ud uespLog.MineRecipeDataStart() from in-game.
- Copy data from the tempData section of the saved variable file, re-format, and paste into esoRecipeData.php.
Sets[edit]
-
- Update the setInfo table with new set types and locations.
- Edit and run generateSetImages.php to create image tooltips if needed.
Styles / Motifs[edit]
-
- This only has to be done if new motifs/styles were added or existing styles changed.
- Use /ud GetHighestItemStyleId() in-game to get the last style ID.
- Use /ud GetItemStyleName(#) in-game to get the style material.
- Use /ud GetItemStyleMaterialLink(#) in-game to get the style material and find the icon name.
- Use /uespstyle all, /uespstyle allid in-game to list all styles.
- Edit uespLog.lua and update the uespLog.CRAFTSTYLENAME_TO_ITEMSTYLE and uespLog.CRAFTSTYLENAME_TO_MOTIFID arrays.
- Edit uespLogTradeData.lua and update the uespLog.ALT_STYLE_ICON_DATA and uespLog.STYLE_ICON_DATA arrays.
- Edit esolog:esoCommon.php and update the $ESO_ITEMSTYLE_TEXTS array.
- Edit esochardata:viewCharData.class.php and update the $MOTIFS_FOR_MASTERWRITS array.
Traits[edit]
-
- This only has to be done when new traits are added.
- Edit esolog:esoCommon.php and update the $ESO_ITEMTRAIT##_FULLTEXTS, $ESO_ITEMTRAIT_DESCRIPTIONS, $ESO_ITEMTRANSMUTETRAIT_IDS, and $ESO_ITEMTRAIT##_TEXTS arrays.
- Edit esolog:resources/esoitemlink.js and update the ESO_ITEM_TRAITS array.
- Edit esolog:resources/esoItemSearchPopup.js and update the UESP.ESO_TRAIT_TYPES array (and select lists if needed).
Antiquities[edit]
-
- Usually only done for live updates.
- Run /ud uespLog.MineAntiquities() in-game and upload/parse the log file.
Other Data[edit]
-
- esolog:esoCommon.php $ESO_ITEMTYPE_TEXTS
- esolog:esoCommon.php $ESO_ITEMSPECIALTYPE_TEXTS and $ESO_ITEMSPECIALTYPE_RAW_TEXTS
-
- Some constants can be checked by running the /ud uespLog.MineApiConstants() command and then extracting the text from the tempData section of the saved variables file.
- esolog:esoCommon.php $ESO_SET_INDEXES (from Sets.php created by the createNewExport.sh script)
Update Icons[edit]
-
- Upload art to content3:
rsync -avz ./esomnf-XX/esoui/art/ dave@content3.uesp.net:/home/uesp/www/eso/gameicons/esoui/art/ rsync -avz ./gamemnf-XX/esoui/art/ dave@content3.uesp.net:/home/uesp/www/eso/gameicons/esoui/art/
-
- Update file permissions:
sudo chown dave:uespadmin -R /home/uesp/www/eso/gameicons/ sudo chmod a+rx,o-w -R /home/uesp/www/eso/gameicons/
-
- Copy from content3 to files1:
rsync -avz --checksum /home/uesp/www/eso/gameicons/ dave@files1.uesp.net:/shared/uesp/esogameicons/
Update API[edit]
-
- This is only done for Live updates.
- Ensure your file system is setup with case-insensitive filenames. (see below for Windows).
- Modify EsoParseData.py and update the INPUT_GLOBAL_FILENAME, INPUT_LUA_PATH, and OUTPUT_PATH variables.
- Run EsoParseData.py from a CygWin shell window.
- If script aborts in error fix the folder security permissions in Windows and retry.
- Edit makeApiDiff.sh to the current and previous API versions and run.
- Upload the new API directory to content2:/home/uesp/www/eso/data/.
- Copy api.html to the root index.html.
- Replace the symbolic link current to the new API directory.
ln -sfn 100025 current
Update Sales Items[edit]
-
- Only done for Live updates.
- Once items have been mined and are live run the script:
php fixNewSalesItems.php
-
- to update the data of any new items that have appeared in sales data since the update.
TODO[edit]
Notes[edit]
-
- Modify Windows to use case insensitive filenames in Cygwin shell.
-
- Install the Linux Subsystem in Windows.
- Run the command:
fsutil.exe file setCaseSensitiveInfo "full path of folder" enable
-
-
- This doesn't change any existing sub-folders but should enable it on any new sub-folders created after the command is run.
- Old Method (No Longer Works) As of ~2018 the below method seems to no longer work.
-
- Set Windows registry key HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\Session Manager\kernel\obcaseinsensitive=dword:00000000.
- Reboot after modifying registry key.
- Use Cygwin with posix=1 set for all relevant drives in /etc/fstab.
- Test case insensitivity before running script.
- (Optional) Set Windows registry key obcaseinsensitive back 1 after finished.
-