Choose timezone
Your profile timezone:
IT-DB-SAS Hadoop Technical Forum
Description
NXCals HBase migration scenarios discussion and other Hadoop service related matters.
Participants:
- Luna (L), Emil (E), Riccardo (R), Manuel (M), Miltiadis (MG)
E: Quickly presenting the agenda
------- Migration scenarios for the NXCALS HBase cluster
E: Quick recap of the migration scenarios up to now
1st option: Parallel Clusters (But we do not get the advantages of the isolation)
2nd option: Zookeeper sharing.
M: Stressing the problem of updating client configs
E: Is more in favor of the 1st option. Arguing that using one zookeeper for more services has the potential risk of overloading it.
M: Getting back to the issue of updating the client configs
R: If it is only for application querying data, they can always just restart the application. It might be easier than expected.
L: There is a need to set a date as an ultimate deadline for the clients to have accepted the new configs
E: We have to make sure the NXCALS team can parallelize the ETLs. If this is a blocker we can exclude this scenario (1st)
M,L: Raising the issue of number of files/data that could potentially go up after the start of operations, exceeding by a lot current forecasts.
E,M: Maybe the NXCALS from their side could implement some filtering to not push all the data both to HBASE and HDFS , arguing for cases that HBASE could be skipped
L: Summarizing the action plan:
1. Update the "News" with the current assets we have (new machines, data storage etc.)
2. Have a first session where we gather information and present the news and a second session (monthly meeting) where we go through presenting the full fledged scenarios.
Of course in the first session we will probably hint a bit our two scenarios (we have to get information from them before we really shape the possible scenarios and be able to fully present them)
M: Highlighting that the risk of the setup should not leave us vulnerable to a scenario were we will be responsible of a failure/incident.
L: Summarizing the questions we need to ask
1. Configuration related questions
2. How much space we have (or expected to require)
3. Two ETLs ( maybe not ask it but present it in a way of being a requirement, or present it along with the advantages is offers)
M: Considering the parallel clusters only alongside a deadline of updating all the client configs and going through the full isolation
------- Decommissioning of old hardware from HDP clusters
E: 4 Machines were to be decommissioned, 3 in trash already 4th one decommissioned now
Next batch deadline until 31 of October, problem lies with the namenodes
M: Presenting the related Jira Ticket
ITHDP1201 - lxhadoop_prod
All: Discussing possilibity of staying only with one namenode
E,M : Promoting another datanode to a namenode, also trying this operation for the first time
Similar problem with ITHDP1301, ITHDP1401 hdpqa_prod
E: Go with VM solution on openstack replacing those two baremetals machines. Issue with how much memory we can request on openstack
Available 256G of memory , Used 32GB
M: Create intervention , announce Monday and start a week after (analytix_prod)
Canali DEV --- Deleting data , making sure that we do not have any client configs references (Most probably we do not really have)
Hdpqa_prod --- Move everything to openstack VMs - We have to announce it now since we have until the end of October
------- Planning intervention on RJ02-30 power source impacting HDP services
E: Should we have an impact action for that ?
L: We should at least inform them in advance, but since it is not our intervention no need for impact
E: In the past they were asking us to open IMPACT on an intervention that they were making
L: Action plan steps:
1. Write email to clarify scenarios with infrastructure team - Setup a meeting with them this afternoon
--- If they do not have required information we must talk with CF
2. OTG for analytix stating the impact
3. Speak with NXCALS and following similar procedure (IMPACT etc.)
There are minutes attached to this event.
Show them.