Hi @chrscheuer,
Is there a way to combine the the scripts below to simulate a mouse drag from a UI elements relative position to an predefined x,y position?
sf.mouse.simulateDrag({
startPosition: {"x":290,"y":10},
endPosition: {"x":290,"y":28},
});
sf.ui.proTools.mainWindow.groups.allItems[7].mouseClickElement({
relativePosition: {"x":290,"y":10},
anchor: "TopLeft",
});
Linked from:
Christian Scheuer @chrscheuer2020-04-18 22:39:25.714ZHi Kitch!
Yes instead of using mouseClickElement, access the element's frame property, from which you can get the screen coordinates to use as input for the simulateDrag event.
However, depending on what you're trying to do it looks like there might be a more accurate solution. Right now you're fetching groups in the mainWindow by number for example - that won't be stable.
Can you describe in more detail what you're trying to do?
Kitch Membery @Kitch2020-04-18 22:48:21.676ZHi mate... I hope you took yesterday off at least!?
In it's simplest form I'm trying to drag track "Insert A" to track "Insert B". (As a part of a larger script to shuffle around inserts.) I'll eventually adapt this to a reusable function (and possible surface).
Rock on!
- In reply tochrscheuer⬆:
Kitch Membery @Kitch2020-04-18 23:18:01.848Z2020-04-19 11:18:47.690ZUsing the elements frame property works great... is there a way to make it work faster?
var trackFrame = sf.ui.proTools.mainWindow.groups.allItems[7].frame; sf.mouse.simulateDrag({ startPosition: {"x":trackFrame.x+290,"y":trackFrame.y+10}, endPosition: {"x":trackFrame.x+290,"y":trackFrame.y+28}, });
Kitch Membery @Kitch2020-04-18 23:19:34.463Z2020-04-18 23:26:10.278ZActually I think this is a Pro Tools limitation as I just tried it manually and noticed how slow it is by default depending on the plugin.
Christian Scheuer @chrscheuer2020-04-20 08:39:54.855ZThere are many ways to optimize this :)
Instead of using the
simulateDragyou can manually perform the simulation by using themouse.downmouse.dragandmouse.upmethods in sequence.But more importantly, don't use relative positioning within the track at all. This will be highly unstable.
Instead, get the frames from the buttons themselves:var insert1BtnFrame = sf.ui.proTools.selectedTrack.insertButtons[0].frame; var insert2BtnFrame = sf.ui.proTools.selectedTrack.insertButtons[1].frame; //...
Kitch Membery @Kitch2020-04-20 09:08:21.619ZGreat... Thanks mate! I'll make some changes in the script in my other post..
Christian Scheuer @chrscheuer2020-04-20 10:26:27.513ZFor making it work faster, the crucial element is to figure out the timing of the events so that PT registers it.
ThesimulateDragfunction is designed to be slow enough to simulate how a human would move the mouse (and likely err on the slow side).If you use mouse.down, mouse.drag and mouse.up, the crucial point is the drag. As a minimum you need to fire off these events:
- mouse.down at the start position
- (Optional:) mouse.drag at points along the route
- mouse.drag at the end position
- sf.wait to make sure PT has registered you're hovering over a valid drop target
- mouse.up at the end position
Particularly the mouse.drag at the end position seems important. You might be able to get away with just a single mouse.drag call at the end position, and then a sf.wait so PT registers that the mouse is being dragged over a valid drop target, and then fire the mouse.up event.
You can experiment with your own versions of this until you find some timing that's more optimized for this scenario.
Kitch Membery @Kitch2020-04-20 11:07:48.938ZThat makes perfect sence and is a good lesson on speeding up scripts.
I think I need a little extra help with this though. I've never used the mouse.down, mouse.drag, mouse.up events before...
How would I do a simple drag insert1BtnFrame to insert2BtnFrame.
This is what I have so far (BTW is not optimised or working for that matter. :-)
//Activate Pro Tools sf.ui.proTools.appActivateMainWindow(); var insert1BtnFrame = sf.ui.proTools.selectedTrack.insertButtons[0].frame; var insert2BtnFrame = sf.ui.proTools.selectedTrack.insertButtons[1].frame; sf.mouse.down({ position: { x: insert1BtnFrame.x+4, y: insert1BtnFrame.y+35 } }); sf.mouse.drag({ position: { x: insert2BtnFrame.x+4, y: insert2BtnFrame.y+35 } }); sf.wait({intervalMs:1000}); sf.mouse.up({ position: { x: insert2BtnFrame.x+4, y: insert2BtnFrame.y+35 } });
Christian Scheuer @chrscheuer2020-04-20 11:47:52.102ZInstead of writing this directly, write it as a function that essentially takes the same arguments as simulateDrag.
Something like this is what I would have in mind:
/** * @param {PointF} startPos * @param {PointF} endPos */ function fastDrag(startPos, endPos) { const numberOfPointsInBetween = 3; const timeToWaitBetweenEachDrag = 100; const timeToWaitBeforeDropping = 500; sf.mouse.down({ position: startPos }); for(var i=0; i<numberOfPointsInBetween; i++) { var dragPos = { x: (endPos.x - startPos.x) / numberOfPointsInBetween * i + startPos.x, y: (endPos.y - startPos.y) / numberOfPointsInBetween * i + startPos.y, }; sf.wait({ intervalMs: timeToWaitBetweenEachDrag }); sf.mouse.drag({ position: dragPos }); } sf.mouse.drag({ position: endPos }); sf.wait({ intervalMs: timeToWaitBeforeDropping }); sf.mouse.up({ position: endPos }); }
Kitch Membery @Kitch2020-04-20 12:09:14.306ZSensational!
And It's 5am... oops!
Christian Scheuer @chrscheuer2020-04-20 15:33:11.622ZHaha :)
- In reply tochrscheuer⬆:
Kitch Membery @Kitch2020-04-21 01:39:21.703Z2020-04-21 21:36:40.380ZI'm getting so close with this one @chrscheuer,
However, intermittently the mouse.down event sometimes toggles the insert open/closed rather than moving it. I'm not sure why though. Any thoughts?
insertSlotSwitcher('Insert A', 'Insert B'); function insertSlotSwitcher(originSlot, destinationSlot) { var insertButtonArray = ['Insert A', 'Insert B', 'Insert C', 'Insert D', 'Insert E', 'Insert F', 'Insert G', 'Insert H', 'Insert I', 'Insert J']; function fastDrag(startPos, endPos) { const numberOfPointsInBetween = 3; const timeToWaitBetweenEachDrag = 100; const timeToWaitBeforeDropping = 1; sf.mouse.down({ position: startPos }); for (var i = 0; i < numberOfPointsInBetween; i++) { var dragPos = { x: (endPos.x - startPos.x) / numberOfPointsInBetween * i + startPos.x, y: (endPos.y - startPos.y) / numberOfPointsInBetween * i + startPos.y, }; sf.wait({ intervalMs: timeToWaitBetweenEachDrag }); sf.mouse.drag({ position: dragPos }); } sf.mouse.drag({ position: endPos }); sf.wait({ intervalMs: timeToWaitBeforeDropping }); sf.mouse.up({ position: endPos }); } function getFirstFreeInsertIndex() { var btns = sf.ui.proTools.selectedTrack.invalidate().insertButtons; for (var i = 0; i < 10; i++) if (btns[i].value.invalidate().value === "unassigned") return i; log("Script failed - There are no free insert slots"); throw 0; } //Establish Index & Frame for First Free Slot var firstFreeSlotIndex = getFirstFreeInsertIndex(); var firstFreeSlotFrame = sf.ui.proTools.selectedTrack.insertButtons[firstFreeSlotIndex].frame; //Establish Index & Frame for Insert Origin var originSlotIndex = insertButtonArray.indexOf(originSlot); var originSlotFrame = sf.ui.proTools.selectedTrack.insertButtons[originSlotIndex].frame; //Establish Index & Frame for Insert Destination var destinationSlotIndex = insertButtonArray.indexOf(destinationSlot); var destinationSlotFrame = sf.ui.proTools.selectedTrack.insertButtons[destinationSlotIndex].frame; //If originSlot is unassigned cancel script if (sf.ui.proTools.selectedTrack.insertButtons[originSlotIndex].value.value === "unassigned") { log("There is no insert on \"" + originSlot + "\""); throw (0); } //Perform Insert Moves if (sf.ui.proTools.selectedTrack.insertButtons[destinationSlotIndex].value.value === "unassigned") { //Move Origin Slot to Sestination Slot fastDrag(originSlotFrame, destinationSlotFrame); } else { //Move Destination Slot to First Free Slot fastDrag(destinationSlotFrame, firstFreeSlotFrame); //Wait for for First Free Slot to be assigned while (sf.ui.proTools.selectedTrack.insertButtons[firstFreeSlotIndex].value.invalidate().value === "unassigned") { sf.wait({intervalMs:1}); } //Move Origin Slot to Sestination Slot fastDrag(originSlotFrame, destinationSlotFrame); //Wait for for Destination Slot to be assigned while (sf.ui.proTools.selectedTrack.insertButtons[destinationSlotIndex].value.invalidate().value === "unassigned") { sf.wait({intervalMs:1}); } //Move First Free Slot to Origin Slot fastDrag(firstFreeSlotFrame, originSlotFrame); }; }Also... I'm not sure I understand what the "numberOfPointsInBetween" is doing.
Thanks in advance. :-)
Christian Scheuer @chrscheuer2020-04-21 11:27:17.642ZI think the reason for that is that there elapses too long time after your initial mouse.down until Pro Tools registers a drag, so it becomes registered as a click.
Try changing
fastDragto this:function fastDrag(startPos, endPos) { const numberOfPointsInBetween = 3; const timeToWaitBeforeFirstDrag = 1; const timeToWaitBetweenEachDrag = 100; const timeToWaitBeforeDropping = 1; sf.mouse.down({ position: startPos }); for (var i = 0; i < numberOfPointsInBetween; i++) { var dragPos = { x: (endPos.x - startPos.x) / numberOfPointsInBetween * i + startPos.x, y: (endPos.y - startPos.y) / numberOfPointsInBetween * i + startPos.y, }; sf.wait({ intervalMs: i == 0 ? timeToWaitBeforeFirstDrag : timeToWaitBetweenEachDrag }); sf.mouse.drag({ position: dragPos }); } sf.mouse.drag({ position: endPos }); sf.wait({ intervalMs: timeToWaitBeforeDropping }); sf.mouse.up({ position: endPos }); }
Kitch Membery @Kitch2020-04-21 22:16:56.677ZThanks for that update to the fastDrag function. :-)
I noticed a few coding errors in my previous code and fixed them... I'd omitted a few ";" at the end of the fastDrag moves.
It seems to be more stable now but I still need to fully test it.
Christian Scheuer @chrscheuer2020-04-22 11:08:30.902ZYou might also need something else. The initial drag might have to be at least a certain amount of pixels for it to be registered as a drag. Worth noting if you're still seeing issues.
Kitch Membery @Kitch2020-04-22 18:23:57.098ZIt seems to be working ok now. Will test it more and see how it goes.
Here is the updated script that works on all selected tracks :-)
sf.ui.proTools.mainWindow.invalidate() sf.ui.proTools.selectedTracks.trackHeaders.slice().map(track => { track.trackSelect(); insertSlotSwitcher('Insert D', 'Insert J'); }); function insertSlotSwitcher(originSlot, destinationSlot) { var insertButtonArray = ['Insert A', 'Insert B', 'Insert C', 'Insert D', 'Insert E', 'Insert F', 'Insert G', 'Insert H', 'Insert I', 'Insert J']; function fastDrag(startPos, endPos) { const numberOfPointsInBetween = 3; const timeToWaitBeforeFirstDrag = 1; const timeToWaitBetweenEachDrag = 100; const timeToWaitBeforeDropping = 1; sf.mouse.down({ position: startPos }); for (var i = 0; i < numberOfPointsInBetween; i++) { var dragPos = { x: (endPos.x - startPos.x) / numberOfPointsInBetween * i + startPos.x, y: (endPos.y - startPos.y) / numberOfPointsInBetween * i + startPos.y, }; sf.wait({ intervalMs: i == 0 ? timeToWaitBeforeFirstDrag : timeToWaitBetweenEachDrag }); sf.mouse.drag({ position: dragPos }); } sf.mouse.drag({ position: endPos }); sf.wait({ intervalMs: timeToWaitBeforeDropping }); sf.mouse.up({ position: endPos }); } function getFirstFreeInsertIndex() { var btns = sf.ui.proTools.selectedTrack.invalidate().insertButtons; for (var i = 0; i < 10; i++) if (btns[i].value.invalidate().value === "unassigned") return i; } function checkForFreeInsert() { if (getFirstFreeInsertIndex() == undefined) return false else return true } if (checkForFreeInsert()) { //Establish Index & Frame for First Free Slot var firstFreeSlotIndex = getFirstFreeInsertIndex(); var firstFreeSlotFrame = sf.ui.proTools.selectedTrack.insertButtons[firstFreeSlotIndex].frame; //Establish Index & Frame for Insert Origin var originSlotIndex = insertButtonArray.indexOf(originSlot); var originSlotFrame = sf.ui.proTools.selectedTrack.insertButtons[originSlotIndex].frame; //Establish Index & Frame for Insert Destination var destinationSlotIndex = insertButtonArray.indexOf(destinationSlot); var destinationSlotFrame = sf.ui.proTools.selectedTrack.insertButtons[destinationSlotIndex].frame; } //Perform Insert Moves if (checkForFreeInsert() === false) { //If no free slots are available log("There are no free insert slots on this track"); } else if (sf.ui.proTools.selectedTrack.insertButtons[destinationSlotIndex].value.value === "unassigned") { //If Destination slot is unassigned //Move Origin Slot to Sestination Slot fastDrag(originSlotFrame, destinationSlotFrame); } else if (sf.ui.proTools.selectedTrack.insertButtons[originSlotIndex].value.value === "unassigned") { //If there is no Insert at the origin log("There is no insert on \"" + originSlot + "\""); } else { //Move Destination Slot to First Free Slot fastDrag(destinationSlotFrame, firstFreeSlotFrame); //Wait for for First Free Slot to be assigned while (sf.ui.proTools.selectedTrack.insertButtons[firstFreeSlotIndex].value.invalidate().value === "unassigned") { sf.wait({ intervalMs: 1 }); } //Move Origin Slot to Sestination Slot fastDrag(originSlotFrame, destinationSlotFrame); //Wait for for Destination Slot to be assigned while (sf.ui.proTools.selectedTrack.insertButtons[destinationSlotIndex].value.invalidate().value === "unassigned") { sf.wait({ intervalMs: 1 }); } //Move First Free Slot to Origin Slot fastDrag(firstFreeSlotFrame, originSlotFrame); }; }
Christian Scheuer @chrscheuer2020-04-22 18:55:52.944ZThat's great!
Noticing a slight issue here. You are using the.frameproperties which are rectangles (ie. they both have topleft origin point as well as the width+height) and you give these to the fastDrag function to use, which expects to receive points. This works now because Rects have x and y properties, and points do too. But it's a little hacky.
Also, it means you're using the very, very top left corner of each component's frame to do the drag, which may or may not be accurate. Normally you'd offset a couple of pixels on both x and y axis.Since it still makes sense to leave the fastDrag function to receive points (what would it need a rect for), you should convert the frames into points before you pass them into the fastDrag function.
For example define a function
getClickPointthat takes a frame as a parameter and returns a point with the x and y values incremented by 3.function getClickPoint(frame) { return { x: frame.x + 3, y: frame.y + 3, }; }And then when you call fastDrag:
fastDrag(getClickPoint(destinationSlotFrame), getClickPoint(firstFreeSlotFrame));
Kitch Membery @Kitch2020-04-22 19:55:51.217ZAhhh yes... Thanks Christian!
Somthing like this?
var insertFrame = sf.ui.proTools.selectedTrack.invalidate().insertButtons[1].frame; log(insertFrame); function getClickPoint(frame) { frame.x = frame.x + 3; frame.y = frame.y + 3; return frame } log(getClickPoint(insertFrame));
Christian Scheuer @chrscheuer2020-04-22 20:07:05.920ZNot exactly - you're still returning a frame instead of returning a point. A frame has 4 properties - x, y, w, h - a point has 2 - x and y.
See my code above :)
Kitch Membery @Kitch2020-04-22 20:15:35.772ZI see... I did it slightly different but I think it still works the same... (maybe?)
I called the getClickPoint on the startPos and endPos within the fastdrag function.
like this;
function getClickPoint(frame) { frame.x = frame.x + 3; frame.y = frame.y + 3; return frame } function insertSlotSwitcher(originSlot, destinationSlot) { var insertButtonArray = ['Insert A', 'Insert B', 'Insert C', 'Insert D', 'Insert E', 'Insert F', 'Insert G', 'Insert H', 'Insert I', 'Insert J']; function fastDrag(startPos, endPos) { getClickPoint(startPos); getClickPoint(endPos); const numberOfPointsInBetween = 3; const timeToWaitBeforeFirstDrag = 1; const timeToWaitBetweenEachDrag = 100; const timeToWaitBeforeDropping = 1; sf.mouse.down({ position: startPos }); for (var i = 0; i < numberOfPointsInBetween; i++) { var dragPos = { x: (endPos.x - startPos.x) / numberOfPointsInBetween * i + startPos.x, y: (endPos.y - startPos.y) / numberOfPointsInBetween * i + startPos.y, }; sf.wait({ intervalMs: i == 0 ? timeToWaitBeforeFirstDrag : timeToWaitBetweenEachDrag }); sf.mouse.drag({ position: dragPos }); } sf.mouse.drag({ position: endPos }); sf.wait({ intervalMs: timeToWaitBeforeDropping }); sf.mouse.up({ position: endPos }); }
Christian Scheuer @chrscheuer2020-04-22 20:18:29.769ZThis is still wrong:
function getClickPoint(frame) { frame.x = frame.x + 3; frame.y = frame.y + 3; return frame }See my posts above, I am saying the same thing over and over... You should not pass a frame to a function expecting a point. This could break in future versions of SoundFlow.
You are returning a frame. Don't return a frame.
Christian Scheuer @chrscheuer2020-04-22 20:19:31.260ZYou are also modifying an object instead of creating a new one. This is not ideal either.
Now the function is called getClickPoint but it doesn't get a point, it modifies a frame.
So this is going in the wrong direction...
Christian Scheuer @chrscheuer2020-04-22 20:21:05.999ZNow that I read your post again, this is even worse, because you moved the logic into fastDrag. That definitely doesn't make sense.
What I was trying to improve in your code was to keep the fastDrag function only worrying about dragging between two points - keep it that way. You just need to feed it the two correct points, calculated from the frames. Hence my initial post.
Kitch Membery @Kitch2020-04-22 20:51:55.219ZThanks. I get it now.
Sorry if I frustrated you.
Christian Scheuer @chrscheuer2020-04-22 21:01:20.572ZAll good. Long day haha, and been dealing with Stream Decks, which is always pretty awful... :)